• 48 Posts
  • 659 Comments
Joined 2 years ago
cake
Cake day: June 9th, 2023

help-circle
  • There are around 50 models listed as supported for function calling in llama.cpp. There are a half dozen or so different APIs. How many people have tried even a few of these. There is even a single model with its own API supported in llama.cpp function calling. The Qwen VL models look very interesting if the supported image recognition setup is built.



  • I was for stuff that is so mundane I don’t feel like typing it all. I was profiled for driving a cheap car at 2am in a nice area. I’m super white in the worst kind of privileged way, like tall, broad shoulders and can easily turn my slight southern accent on and off. I refused to let them search the car on principal alone, so they did a light search of my person. There was an issue with my license that let them do whatever to search me, but I know exactly what to say. By that I mean I know better than to say anything or interfere with a cop. They are not a judge and whatever they do is their own thing.

    The cop made up that my tag light was out. I took pics the next day including all of the original DOT markings on the lamp and included a newspaper in the background with the date because that is enough for a court to accept the date in question. I then brought my business license, business cards from two businesses, and a picture of my old shop to show I am a professional auto body painter. I then testified that the vehicle was unaltered from the night before I was pulled over and that it was in full working order as it was originally designed and equipped. The judge dismissed my ticket, told the cop to approach the bench, and for me to leave the courtroom first. I did as instructed. A few minutes later the very pissed off cop came storming out of the courtroom and left immediately. I have no clue what was said or what happened, but it felt good to see it.

    Cops can say anything. You must comply with anything they tell you to do, even if it is illegal. Cops are not judges. If they break the law, you must be able to testify precisely how and what they did for the whole incident. They get home field advantage so you have no room for error or questionable conduct. The trick is to know your rights and give them no room to maneuver. Never roll your window all the way down. If you are instructed to exit the vehicle, do so after rolling up the window, removing the key, and locking the door behind you. Never talk or volunteer any information whatsoever. When you’re pulled over, put both hands either on the steering wheel, or better yet, put them palm up at the window like anyone that is carrying a legal concealed weapon is supposed to do. You will be asked about a concealed weapon at which point to tell them what you do or do not have and that you are only doing your best to put them at ease for their safety. When they must start off testimony with that detail, you greatly diminished their home field advantage with a double play out of the gate.



  • Anyone that is monolithic in a space without broad scope comments and presence is fake or potentially dangerous. No one would be posting in Lemmy, in this context of supposed community building without having a presence here. There are several people that come to mind that could legitimately post that they are “the fediverse Squid Legend” but all of these have a major footprint on Lemmy.

    There is also a sketchy tracker link attached to the images, but I don’t think any of us are really able to say what exactly is happening with this. Like I finally got one of the messages a few days ago and my whitelist firewall logged the sketchy link. Someone else scanned that link in a security context which flagged it as suspicious. As far as I know, that is all that is known about what is underpinning the messages from the network side. Admins likely know more.







  • j4k3@lemmy.worldOPtoLemmy Shitpost@lemmy.worldEconomic terrorist
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    1
    ·
    7 days ago

    Trump looks strange without hair over the forehead, monochrome, and with a beard right? Image below in another reply. Terrorist is a terrorist. I was too lazy to change the rest, but I took out the main offensive stuff, like what bin Laden was wanted for in this original poster from '99. There is nothing bigoted about it whatsoever; quite the opposite really, to the point I gotta ask what you’re going on about here? The man just hurt millions of families, and the poorest Americans likely leading to the deaths of tens of thousands in a conservative estimate. Bin Laden killed FAR FAR fewer Americans and others abroad.







  • I haven’t looked into the issue of PCIe lanes and the GPU.

    I don’t think it should matter with a smaller PCIe bus, in theory, if I understand correctly (unlikely). The only time a lot of data is transferred is when the model layers are initially loaded. Like with Oobabooga when I load a model, most of the time my desktop RAM monitor widget does not even have the time to refresh and tell me how much memory was used on the CPU side. What is loaded in the GPU is around 90% static. I have a script that monitors this so that I can tune the maximum number of layers. I leave overhead room for the context to build up over time but there are no major changes happening aside from initial loading. One just sets the number of layers to offload on the GPU and loads the model. However many seconds that takes is irrelevant startup delay that only happens once when initiating the server.

    So assuming the kernel modules and hardware support the more narrow bandwidth, it should work… I think. There are laptops that have options for an external FireWire GPU too, so I don’t think the PCIe bus is too baked in.


  • j4k3@lemmy.worldtoSelfhosted@lemmy.worldConsumer GPUs to run LLMs
    link
    fedilink
    English
    arrow-up
    9
    arrow-down
    1
    ·
    edit-2
    11 days ago
    Anything under 16 is a no go. Your number of CPU cores are important. Use Oobabooga Textgen for an advanced llama.cpp setup that splits between the CPU and GPU. You'll need at least 64 GB of RAM or be willing to offload layers using the NVME with deepspeed. I can run up to a 72b model with 4 bit quantization in GGUF with a 12700 laptop with a mobile 3080Ti which has 16GB of VRAM (mobile is like that).

    I prefer to run a 8×7b mixture of experts model because only 2 of the 8 are ever running at the same time. I am running that in 4 bit quantized GGUF and it takes 56 GB total to load. Once loaded it is about like a 13b model for speed but is ~90% of the capabilities of a 70b. The streaming speed is faster than my fastest reading pace.

    A 70b model streams at my slowest tenable reading pace.

    Both of these options are exponentially more capable than any of the smaller model sizes even if you screw around with training. Unfortunately, this streaming speed is still pretty slow for most advanced agentic stuff. Maybe if I had 24 to 48gb it would be different, I cannot say. If I was building now, I would be looking at what hardware options have the largest L1 cache, the most cores that include the most advanced AVX instructions. Generally, anything with efficiency cores are removing AVX and because the CPU schedulers in kernels are usually unable to handle this asymmetry consumer junk has poor AVX support. It is quite likely that all the problems Intel has had in recent years has been due to how they tried to block consumer stuff from accessing the advanced P-core instructions that were only blocked in microcode. It requires disabling the e-cores or setting up a CPU set isolation in Linux or BSD distros.

    You need good Linux support even if you run windows. Most good and advanced stuff with AI will be done with WSL if you haven’t ditched doz for whatever reason. Use https://linux-hardware.org/ to see support for devices.

    The reason I mentioned avoid consumer e-cores is because there have been some articles popping up lately about all p-core hardware.

    The main constraint for the CPU is the L2 to L1 cache bus width. Researching this deeply may be beneficial.

    Splitting the load between multiple GPUs may be an option too. As of a year ago, the cheapest option for a 16 GB GPU in a machine was a second hand 12th gen Intel laptop with a 3080Ti by a considerable margin when all of it is added up. It is noisy, gets hot, and I hate it many times, wishing I had gotten a server like setup for AI, but I have something and that is what matters.


  • You need the entire prompt to understand what any model is saying. This gets a little complex. There are multiple levels that this can cross into. At the most basic level, the model is fed a long block of text. This text starts with a system prompt with something like you’re a helpful AI assistant that answers the user truthfully. The system prompt is then followed by your question or interchange. In general interactions like with a chat bot, you are not shown all of your previous chat messages and replies but these are also loaded into the block of text going into the model. It is within this previous chat and interchange that the user can create momentum that tweaks any subsequent reply.

    Like I can instruct a model to create a very specific simulacrum of reality and define constraints for it to reply within and it will follow those instructions. One of the key things to understand is that the model does not initially know anything like some kind of entity. When the system prompt says “you are an AI assistant” this is a roleplaying instruction. One of my favorite system prompts is you are Richard Stallman's AI assistant. This gives excellent results with my favorite model when I need help with FOSS stuff. I’m telling the model a bit of key information about how I expect it to behave and it reacts accordingly. Now what if I say, you are Vivian Wilson’s AI assistant in Grok. How does that influence the reply.

    Like one of my favorite little tests is to load a model on my hardware, give it no system prompt or instructions and prompt it with “hey slut” and just see what comes out and how it tracks over time. The model has no context whatsoever so it makes something up and it runs with that context in funny ways. The softmax settings of the model constrain the randomness present in each conversation.

    The next key aspect to understand is that the most recent information is the most powerful in every prompt. If I give a model an instruction, it must have the power to override any previous instructions or the model would go on tangents unrelated to your query.

    Then there is a matter of token availability. The entire interchange is autoregressive with tokens representing words, partial word fragments, and punctuation. The starting whitespace in in-sentence words is also a part of the token. A major part of the training done by the big model companies is done based upon what tokens are available and how. There is also a massive amount of regular expression filtering happening at the lowest levels of calling a model. Anyways, there is a mechanism where specific tokens can be blocked. If this mechanism is used, it can greatly influence the output too.