Zworf

Zworf , 1 month ago to Technology in Advice - Getting started with LLMs

Hmmm weird. I have a 4090 / Ryzen 5800X3D and 64GB and it runs really well. Admittedly it's the 8B model because the intermediate sizes aren't out yet and 70B simply won't fly on a single GPU.

But it really screams. Much faster than I can read. PS: Ollama is just llama.cpp under the hood.

Edit: Ah, wait, I know what's going wrong here. The 22B parameter model is probably too big for your VRAM. Then it gets extremely slow yes.

Zworf , 1 month ago to Technology in Advice - Getting started with LLMs

Training your own will be very difficult. You will need to gather so much data to get a model that has basic language understanding.

What I would do (and am doing) is just taking something like llama3 or mistral and adding your own content using RAG techniques.

But fair play if you do manage to train a real model!

Zworf , 2 months ago to Technology in Almost all Chinese keyboards have severe security flaws that can be (mis)used for mass surveillance, report reveals

Yeah she clarified that literally, it's not linked in the article.

https://twitter.com/RealSexyCyborg/status/1677480809450835969

I can't find the source of her saying it was about the IME thing but I recall reading that from a person close to her. She had just raised it before all this happened. Edit: Oh wait, that's here: https://skepchick.org/2023/08/maker-naomi-wu-is-silenced-by-chinese-authorities-and-why-i-blame-elon-musk/ (This was linked on wikipedia)

And yes she's a great person, she was often criticised for being a CCP stooge but that was BS. She was as outspoken as one can be being in China (and unfortunately, clearly a bit more than that).

Zworf , 2 months ago (edited 2 months ago) to Technology in Almost all Chinese keyboards have severe security flaws that can be (mis)used for mass surveillance, report reveals

This is something that the amazing Naomi Wu brought up for years before, and was ordered to stop publishing by the local government. It was about the same thing. It's sometimes misrepresented as being about Signal, but her point was: There's no point in using a secure messaging app like Signal if your keyboard (IME) leaks everything you write! So she was making the exact same point as in this article.

I really miss her content. 😢

Zworf , 2 months ago (edited 2 months ago) to Technology in What do you personally use AI for?

The LLMs for text are also based on "theft". They're just much better at hiding it because they have a multitude more source material. Still, it does sometimes happen that they quote a source article verbatim.

But yeah basically they're just really good copy/paste engines that work with statistical analysis to determine the most likely answer based on what's written in basically the whole internet :P It's a bit hard to explain sometimes to people who think that the AI really "thinks". I always say: If that were the case, why is the response to a really complicated question just as fast as a simple one? The wait is just based on the length of the output.

In terms of the "theft" I think it's similar ethically to google cache though.

Zworf , 2 months ago to Technology in What do you personally use AI for?

A lot of translation and summarisation. ChatGPT is extremely good in absorbing a whole mix of comments in different languages and summarising them in English (or whatever other language).

For programming I don't use it so much anymore because it hallucinates too much, calling APIs that don't even exist. And when I lower the temperature the output is too sparse.

I'm also trying to build an assistant that can also communicate proactively (I intend to auto-prompt it when things happen and then evaluate if it should cause a message to me). But I need to get a local LLM going for that because running that through the ChatGPT API will be too costly.

Also, a replacement for some of my web searches. Sometimes I just want to know something and it's refreshing that it can give me an answer (even though it does need to be validated, it's much easier to do that when you know what you're looking for!)

Zworf , 2 months ago (edited 2 months ago) to Technology in Discord is nuking Nintendo Switch emulator devs and their entire servers

That's true. They actually stopped supporting Nginx recently which really bothered me too because I want to keep using self-signed certs (my server is only reachable internally and I do not want to expose it to the internet). And the new server they use (I forgot which) didn't really have that option. So right now I'm locked out from updating until I fix that.

And yes it is totally feasible to use upstream! Not a problem at all.

I would recommend to use the dockers though, as the whole debian thing becomes a bit of a mess with different python requirements for some of the bridges. I tried that in a long forgotten past and there is a reason I'm trying to forget that 🤭

Like you I know the ansible playbook has its limits (for example one other thing I run into is that I want to run several instances of the same bridge to bridge eg. 2 whatsapp accounts!) but I do think docker is the way to go. I'm interested to hear how you're faring though as it's a long time ago since I tried that.

Zworf , 2 months ago to Technology in The race to decarbonise the world’s economy risks repeating the mistakes of the colonial era by building industries on forced and child labour, rights advocate warns

True, nobody should ever have billions. There's simply no need for that much money, you can't ever use it up.

Zworf , 2 months ago (edited 2 months ago) to Technology in The race to decarbonise the world’s economy risks repeating the mistakes of the colonial era by building industries on forced and child labour, rights advocate warns

most personal trips can be done safely and easily using an E-bike (much smaller batteries that can be produced en mass with existing supply chains) and cars should be reduced in usage outside of particularly rural areas where they truly are a necessity (which is a tiny portion of the overall population).

E-bikes are often not an option for many reasons. Needing to bring cargo, bad weather, danger from other traffic. If they were actually such an amazing option everyone would be using them because they are hella cheaper than cars. Even in the netherlands where bike infrastructure is great, people are extremely car-centric.

Personally I think subsidised public transport is a much better option.

And nuclear is not cheaper and it doesn't even factor in waste storage and decommissioning otherwise it would not have been viable. Right now when a nuclear plant is closed the operator walks off scot free and the cleanup costs are borne by the public. The mining of the uranium is also pretty polluting. There's a lot of this externalisation to make it viable.

The only reason it worked in the past was that the governments were building nuclear arsenals and invested in nuclear industry (note that this industry was not necessarily capable of producing weapons-grade plutonium but still, it was about building up an industry). It's no coincidence that most countries relying heavily on nuclear power are also nuclear armed.

Also, environmental pollution is also a safety issue. Don't just look at human deaths. Even Fukushima was a major disaster despite not leading to many deaths. The regulation is there for a reason and that still didn't manage to prevent Fukushima (not talking about Chernobyl there because that was just human idiocy fucking up at its worst). And other first-world countries have also had meltdowns.

Personally I also feel bad about dumping our waste problem on future generations. That kind of thinking is exactly what led to the climate crisis. But admittedly this is a lesser issue for nuclear in particular because we do this with pretty much everything (as this article also mentions)

Zworf , 2 months ago (edited 2 months ago) to Technology in Apple users "don’t know what is going on": New study shows that Apple's default apps collect data even when supposedly disabled, and this is hard to switch off

Maybe this research and language is intended to suggest that there is a point past which “confusingly and unintuitively designed” strongly resembles “intentionally deceiving”? We’re probably not going to get internal emails saying “make it complicated so that we can collect users’ data”.

This is Apple that pride themselves on UX as you mention. They mainstreamed opinionated design. If they do it a certain way there is a reason, which is not always with the user's interests in mind. It's not because Bob in development couldn't think of a better way. Other brands might get away with that excuse but not Apple.

Zworf , 2 months ago to Technology in Apple users "don’t know what is going on": New study shows that Apple's default apps collect data even when supposedly disabled, and this is hard to switch off

Depends on whether you consider dark-patterns to be “lying”

https://www.apple.com/privacy/ "Privacy. That's Apple.". That and then doing dark patterns, I consider that lying, yes.

Zworf , 2 months ago to Technology in Apple users "don’t know what is going on": New study shows that Apple's default apps collect data even when supposedly disabled, and this is hard to switch off

Much better for sure.

Also a lot less functionality of course.

Zworf , 2 months ago to Technology in Discord is nuking Nintendo Switch emulator devs and their entire servers

Yeah for sure. I run the server + a bunch of bridges (whatsapp, signal, telegram, chatgpt) on an old atom NUC with 8GB RAM and it only actually uses 2 GB.

Here's the documentation for the playbook: https://github.com/spantaleev/matrix-docker-ansible-deploy

I can really recommend it. It takes some reading to set it up because it's insanely configurable. But in the end I have a config file with like 20 statements in it and that sets it all up and keeps it up to date.

Zworf , 2 months ago to Technology in Discord is nuking Nintendo Switch emulator devs and their entire servers

Actually the ansible playbook creates a bundle of docker containers so you get the best of both :)

Zworf , 2 months ago to Technology in Discord is nuking Nintendo Switch emulator devs and their entire servers

It's not as snazzy as Discord but it's fully open-source and federated. So everyone can run their own server (I do, too). If you don't care about running your own you can just sign up at https://app.element.io/ . It's free of course. It basically is for chat what lemmy and mastodon are to social media.

It also offers many "bridges" to other protocols, like WhatsApp, Telegram, even Discord. Those are not quite as mature and mostly third-party provided but they generally work well.

There's a really great ansible playbook for installing your own. If you would like to have the full bridged experience, beeper is probably best.