Leave a Review Private LLM
Amazing!
Can’t believe my iPad is so powerful!! Works a charm on my M1. I download Phi3 no problem. You can also get it talk via clicking on text then speech. I then downloaded another model, which wasn’t show in list of installed models. I had to quit the app and go back into it to see the new models, then voila! [It may seem obvious but worth mentioning, some users may not quit the app, and quick to act in leaving negative feedback.] Can i make a request? Can you add the best model of Aya23 for translations?Show lessTony the Vampire
Developer Response
Thanks for the review! Also thanks for reporting the downloaded model list synchronization issue. We've fixed it and it'll go out with the next update. We'd have loved to add the aya-23-8B model, but sadly it's licensed under a cc-by-nc license making it legally untenable for us to add it. We'll be adding the newer QWen2 models soon, which are liberally licensed and were trained on 29 languages (Aya models were trained on 23 languages). We expect those models to do well on translation tasks.Font Size
The main issue I have is not being able to adjust the Font Size, it's tiny compared to even other apps.
Makes it almost unusable IMO. Great app overall besides that, super fast, responsive, etc.Filterless1
App freezes
Updated review, had to re download the app twice, but now it’s working well. I have to say I’m impressed with the app. It is quick on response, and the models I have tried so far are excellent. The only improvement I could wish for would be the ability to save and reload chats.dropped adobe
Developer Response
Thanks for the review! Conversation history is coming soon. :) The next few updates will be focused on adding more downloadable models, conversation history will in an update come soon after that.Awesome, BUT..
I left the app for not even 2 minutes and when i came back, my long conversation was cleared and i wasn’t done with the conversation yet.
Not having a built in chat history that i can return to is actually really frustrating.ItachiiSoLit (YouTube)
Developer Response
Thanks for the review! iOS unloads apps in the background due to memory pressure. We're working on adding conversation history to the app and it should be released as an update, soon.Room for improvement....
Review of the Mac app and not the LLMs as that's outside of the dev's control.
The app should have the option to close the window but not the app when you click on X.
Model selection should be more intuitive by showing which are the latest.
Better integration / system-wide controls would be awesomeShow lessctrl+alt+dileep
Works as intended.
Wish there was an option for adding additional models, but the ones that are available work well. Now if y’all were able to do a text-to-voice implementation with custom voices that’d be amazing. I’m not particular tech savvy, but the fact this app is capable of working offline is amazing.Balishisk
Why Phi-3 only 4-bit!!!
This app looks ok, but they are ruining the experience by only allowing very low quants of the model. I got this app just to use Llama 3 at 4-bit quant and Phi-3 mini at 8-bit quant (which an iPad Pro M2 can easily handle). Please add these quants or allow a way to add ours. Happy to update the review if this is done.
If you’re worried about the app crashing on older hardware you can add a warning to those models.Show lessshadows_lord
Developer Response
Thanks for the review! We've got a task on our roadmap to allow users to download 4-bit quantized versions of Llama 3 8B on 16GB M1/M2 iPads (and possibly next iPhone 16 Pro/Pro Max when they come out). All Llama 3 8B models on macOS will be 4-bit OmniQuant quantized (update releasing this week). While I agree that 4-bit quants are better than 3-bit quants in perplexity, 8-bit quants are unnecessary with OmniQuant. Your priors seem to be from the llama.cpp/Ollama/LMStudio world where they use RTN (round to nearest) quants and Q4 RTN quantized models aren't great. We invest a lot of GPU time and human effort into quantizing models with OmniQuant and 3-bit quantized OmniQuant models are comparable in perplexity to Q4 RTN quantized models and 4-bit quantized OmniQuant models are comparable to Q8 RTN quantized models. I encourage you to read the OmniQuant paper if you're interested in the details.Not good for coding
The builtin model (StableLM Zephyr 3B 4-bit OmniQuant) is not good at code generation. It doesn't follow instructions closely. When I downloaded WhiteRabbitNeo-13B-v1 4-bit OmniQuant and tried to switch to it, the app crashes.
Another drawback is that it's not possible to have separate chats for the same LLM version. Everything is in a single chat log which doesn't allow you to "configure" the chat bot with certain instructions for different topics.
Also, I can't see a blinking text cursor so I can't easily edit my prompt before submitting it. I have the same issue here in the App Store review textbox, weirdly enough.
The nice thing about this app is that it's private and saves you the hassle of figuring out how to install these chat bots by hand. For £9.99 that's not a bad deal for me, but given the aforementioned limitations I will probably continue to research the manual installation process, also because I wanted to try the new Llama 3 version which this app doesn't seem to support.Show lesstemocj
Developer Response
Thanks for the review and the feedback! I'd recommend trying the Mistral 7B based CodeNinja model for coding. We've already added three 3-bit OmniQuant quantized versions of Llama 3 8B based models to the iOS app (universal app, thus a free download for anyone who's purchased the macOS app), we'll release 4-bit OmniQuant quantized versions of the same on the macOS app, next week. Also, we'll be adding support for multiple conversations, soon.Phi3-Mini Please! (Nice app even on Intel Mac)
I'm reading Phi3-Mini is quite capable for its size. I know it was just released, but if you could get a version up soon that would be great. Maybe your existing Phi2 implementation will facilitate that.
Easy to download and run an LLM locally. No muss, no fuss.
Would like to echo another reviewer: please add support for increasing font size. I found I can actually access the default Mac font menu and specify something different, but it doesn't seem to have any effect on the actual font used in the application.Show lessPaul, just Paul ...
Developer Response
Thanks for the review! We’d added support for downloading Phi3-Mini the day it was released, on the iOS app (universal app, thus a free download for anyone who's purchased the macOS app). We’ll add it to the macOS app, along with the three Llama 3 8B based models that are already on the iOS app, next week. Also, I understand the need for adjustable font sizes on macOS. On iOS, people use Dynamic Type to globally alter font sizes from Settings, but macOS doesn’t have Dynamic type (yet). We’ll prioritize this on our roadmap. EDIT: Support for Phi-3 and Llama 3B models was shipped in the latest update (v1.8.5).Basic but it works
Pros:
-simple interface
-If you are looking to run language models on your own device this does the job
-lots of models you can choice from
-you can run llama-3 8B on an iPhone with 6GB of ram (wow)
Cons:
-occasional performance problems which are fixed with a restart
-as of now the app is very barebones in features especially for those looking to tinker
-very aggressive quants mean a drop in output quality but this is due to hardware restrictions from the limited amount of RAM available on mobile devices
-some interesting models are unavailable to download
Overall I recommend this app, just don’t expect the same quality, speed or features of an online monthly subscription llm app like ChatGPT. Looking forward to future updates from the developers!Show lessDevlist
