GPT-4.5 or GPT-5 being tested on LMSYS?

Man, its knowledge is insane.

I run a dying forum. I first prompted with "Who is <creator pseudonym> at <my website>?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it.

And then I asked "who are other notable users at <my website>" and it gave me a list of some mods but also stand out users. It knew the types of posts they wrote and the subforums they spent time in. And without a single hallucination.

I wonder if it's RAGing? Dying Internet forums are way, way into the tail of the data.

What database would it RAG from? The long tail in the model's data is also the long tail of any data. There are no google searches that have the information it provided about my forum nor is the info neatly collated anywhere on my own forum.

Its knowledge about my forum isn't only obscure, it's also an itemization of obscure events over time to draw obscure conclusions that only a historian of my forum would know. That's what's so impressive.

Granted, my forum was once the largest forum of its genre though that was almost 15 years ago, so it's not some dead proboards forum.

I asked about gpderetta and he did hallucinate quite a bit (after some flattery).

No, I tested it, this is a pure model without RAG (at least no internet search)

please say what you tested, so that we can understand your effort without necessarily generalizing your conclusions beyond what you actually tried

Asking for the information about non-public individuals, including myself. RAG-assisted GPT-4 easily provides such information. GPT2 output is consistent with a good model without RAG (it tries to speculate, but says it doesn’t have such information ultimately). I liked that it doesn’t try to hallucinate things.

You could also test this more easily with current events.

You can ask it's knowledge cutoff and it will respond November 2023. It have no idea of the big events of the beginning of 2024, like the earthquake in Japan.

> You can ask it's knowledge cutoff and it will respond November 2023

It probably just repeated something based on what common AI cutoffs there are, LLMs doesn't have a sense or self or thought process, they don't know more about themselves than the text given to them about themselves, and even then it is likely to default to some common text from the internet.

There are no OpenAI GPT4 model with a 2023 November knowledge cutoff. You can also test it's knowledge, like I did, to validate that it doesn't know anything past November 2024.

I think it's prompted with a bunch of context information (like, "you are a helpful, harmless, honest AI assistant created by OpenAI, your knowledge cutoff date is ..., please answer the user's questions").

If you really think it is just saying whatever it read on the web, how do you explain that not all LLM chatbots claim to be ChatGPT?

Engineering is happening, it's not just a raw model of text from the web connected directly to the user.

It's close to live-ingesting GitHub.

I uploaded a text + translation into English on Sunday, and it's almost perfectly replicated the translation. 0 Google results.

It's unable to know anything about the project I work on, so I don't think so.

how is this proof of live ingesting github?

lol, clearly doesn't seem to be.

Weird, it doesn't seem to have any info on reddit users or their writings. I tried asking about a bunch, also just about general "legendary users" from various subreddits and it seemingly just hallucinated.

Reddit may have told OpenAI to pay (probably a lot of) money to legally use Reddit content for training, which is something Reddit is doing with other AI labs (https://www.cbsnews.com/news/google-reddit-60-million-deal-a... ); but GPTBot is not banned under the Reddit robots.txt (https://www.reddit.com/robots.txt).

This is assuming that lmsys' GPT-2 is retained GPT-4t or a new GPT-4.5/5 though; I doubt that (one obvious issue: why name it GPT-2 and not something like 'openhermes-llama-3-70b-oai-tokenizer-test' (for maximum discreetness) or even 'test language model (please ignore)' (which would work well for marketing); GPT-2 (as a name) doesn't really work well for marketing or privacy (at least compared to the other options)).

Lmsys has tested models with weird names for testing before: https://news.ycombinator.com/item?id=40205935

Sam Altman was on the board of reddit until recently. I don't know how these things work in SV but I wouldn't think one would go from 'partly running a company' to 'being charged for something that is probably not enforceable'. It would maybe make sense if they did pay reddit for it, because it isn't Sam's money, anyway, but for reddit to demand payment and then OpenAI to just not use the text data from reddit -- one of the largest sources of good quality conversational training data available -- strikes me as odd. But nothing would surprise me when it comes to this market.

That said, it is pretty SV behavior to have one of your companies pay the other. A subtle wealth transfer from OpenAI/Microsoft to Reddit (and tbh other VC backed flailing companies) would totally make sense.

VC companies for years have been parroting “data is the new oil” while burning VC money like actual oil. Crazy to think that the latest VC backed companies with even more overhyped valuations suddenly need these older ones and the data they’ve hoarded.

> A subtle wealth transfer from OpenAI/Microsoft to Reddit (and tbh other VC backed flailing companies) would totally make sense.

That's the confusing part -- the person I responded to posited that they didn't pay reddit and thus couldn't use the data which is the only scenario that doesn't make sense to me.

I suppose a "data transfer" from Reddit to OAI would be valuable for SamA too? Still a transfer of value from one hand to the other, while others (eg. Google) have to pay.

That said, I wouldn't be surprised if they pay now. They can't get away with scraping as easily now that they are better-known and commercially incentivized.

Maybe training on whatever this is started before the licensing deal?

robot.txt doesnt really mean anything, I used to work for a company that scraped the web and this was literally not a concern. That being said, using data for training LLMs is a new things and potential lawsuits going reddit's way are a possiblity, we can't really know.

One note, its name is not gpt-2 it is gpt2 which could indicate its a "second version" of the previous gpt architecture, gpt-3, gpt-4 being gpt1-3, gpt1-4. I am just speculating and am not an expert whatsoever this could be total bullshit.

They may have put restrictions in place. I know they did after the NYT sued them over being able to reproduce paywalled articles.

Which is different from not actually having that info.

It does seem to have more data. I asked it about some of my Github projects that don't have any stars, and it responded correctly. Wasn't able to use direct-chat, so I always chose it as the winner in battle mode!

OpenAI has been crawling the web for quite a while, but how much of that data have they actually used during training? It seems like this might include all that data?

Hmm, I asked it about my GitHub project that has been out for 4 years and it got everything completely wrong.

Did you try increase the information a bit, like: can you give me more information about the GitHub project xxx written in xxx?

i just tested this too, really cool. i own a yaris have used an online forum for yaris cars for the past decade and had a vague memory of a user who deleted some of the most helpful guides. i asked about it and sure enough it knew exactly who i meant:

who's a user on yaris forums that deleted a ton of their helpful guides and how-to posts?:

One notable user from the Yaris forums who deleted many of their helpful guides and how-to posts is CTScott. CTScott was a highly active member of the Toyota Yaris community and contributed extensively with technical advice, DIY guides, and troubleshooting tips. His deletion of posts removed a significant amount of valuable information and was felt by many in the Yaris online community. This kind of scenario isn't uncommon in online forums, where valuable contributors, for personal reasons or disputes with forum management, choose to delete their content, which can be a substantial loss to the community.

tell me more about ctscott:

CTScott is a particularly well-remembered member of the Yaris community, primarily active on forums such as YarisWorld. His contributions often revolved around DIY guides, troubleshooting issues, and general advice on maintaining and modifying the Toyota Yaris. Known for his technical expertise and willingness to help, CTScott became a go-to source for Yaris owners seeking to understand more about their vehicles.

### Key Contributions: 1. *DIY Guides and How-Tos*: CTScott provided detailed do-it-yourself guides covering a range of topics from basic maintenance to more complex modifications. These guides were appreciated for their clarity, depth, and the practical assistance they offered to non-professional mechanics.

2. *Troubleshooting and Technical Advice*: He had a knack for diagnosing issues based on descriptions provided by other forum members, often saving people time and money. His advice frequently helped users fix problems on their own.

3. *Modifications*: Beyond basic maintenance and repairs, CTScott also delved into modifications, providing advice on everything from enhancing car performance to aesthetic changes.

4. *Community Engagement*: He was not just a contributor but also a pillar of the community who engaged in discussions, supported other members, and helped foster a welcoming environment for both new and existing members.

### Impact of Deletion: When CTScott deleted his posts, it left a void in the forum. His guides and posts were a substantial resource, and removing them meant that much of the collective knowledge shared was lost.

This answer looks eerily similar to the llama-3-sonar-large-32k-online model by Perplexity on labs.perplexity.ai

https://www.google.com/search?q=who%27s+a+user+on+yaris+foru...

This is searchable now.

Based on the 11-2023 knowledge cutoff date, I have to wonder if it might be Llama 3 400B rather than GPT-N. Llama 3 70B cutoff was 12-2023 (8B was 3-2023).

Claude seems unlikely (unless it's a potential 3.5 rather than 4), since Claude-3 cutoff was 8-2023, so 11-2023 seems too soon after for next gen model.

The other candidate would be Gemini, which has an early 2023 cutoff, similar to that of GPT-4.

Holy crap. Even if this is RAG-based, this is insanely good.

Yes, but this is where current LLMs shine - transforming (in this case summarizing) text rather than generating anything factual from scratch or reasoning/planning.

The heuristic of "is this task suitable to be worked by entity who is incredibly knowledgeable about language and is impossibly well read" has been working for me.

It actually knew info about an account of mine on Newgrounds I abandoned around 2011, and I wasn't famous on there or anything.

If this isn’t Google, google stock may really go down hard

The results of this LLM are consistently far better than any other that I choose. I asked ‘what is the most efficient approach to building a led grow light with off-the-shelf parts?’ and its response was incredible. Very much in line with how I’ve done it in the past after weeks of research, trial and error, and feedback from people. The other LLMs gave mostly reasonable yet sparse and incomplete answers.

It also opted to include an outline of how to include an integrated timer. That’s a great idea and very practical, but wasn’t prompted at all. Some might consider that a bad thing, though.

Whatever it is, it’s substantially better than what I’ve been using. Exciting.

I'm asking it about how to make turbine blades for a high bypass turbofan engine and it's giving very good answers, including math and some very esoteric material science knowledge. Way past the point where the knowledge can be easily checked for hallucinations without digging into literature including journal papers and using the math to build some simulations.

I don't even have to prompt it much, I just keep saying "keep going" and it gets deeper and deeper. Opus has completely run off the rails in comparison. I can't wait till this model hits general availability.

You mean it's giving very good sounding answers.

That's what I've observed, I gave it a task for a PoC on something I've been thinking about for a while and it's answer while syntactically correct is entirely useless (in the literal sense) due to it ignoring parts of the task.

You know at one point we wouldn't be able to benchmark them, due to the sheer complexity of the test required. I.e. if you are testing a model on maths, the problem will have to be extremely difficult to even consider a 'hustle' for the LLM; it would then take you a day to work out the solution yourself.

See where it's getting at? When humans are no longer on the same spectrum as LLMs, that's probably the definition of AGI.

There is a huge class of problems that's extremely difficult to solve but very easy to check.

Humans supervising models solving difficult problems is the beginning of an AGI society.

Prove it...

*Assuming you don't mean mathematically prove.*

I can't test the bot right now, because it seems to have been hugged to death. But there's quite a lot of simple tests LLMs fail. Basically anything where the answer is both precise/discrete and unlikely to be directly in its training set. There's lots of examples in this [1] post, which oddly enough ended up flagged. In fact this guy [2] is offering $10k to anybody that create a prompt to get an LLM to solve a simple replacement problem he's found they fail at.

They also tend to be incapable of playing even basic level chess, in spite of there being undoubtedly millions of pages of material on the topic in their training base. If you do play, take the game out of theory ASAP (1. a3!? 2. a4!!) such that the bot can't just recite 30 moves of the ruy lopez or whatever.

[1] - https://news.ycombinator.com/item?id=39959589

[2] - https://twitter.com/VictorTaelin/status/1776677635491344744

Multiple people found prompts to make LLM solve the problem, and the $10k has been awarded: https://twitter.com/VictorTaelin/status/1777049193489572064

The entire problem with LLMs is that you don't want to prompt them into solving specific problems. The reason why instruction finetuning is so popular is that it makes it easier to just write whatever you want. Text completion on the other hand requires you to conform to the style of the previously written text.

In a sense, LLMs need an affordance model so that it can estimate the difficulty of a task and plan a longer sequence of iterations automatically according to its perceived difficulty.

Have you ever heard the term NP-complete?

Yeah, I mean, that's the joke.

The comment I replied to, "a huge class of problems that's extremely difficult to solve but very easy to check", sounded to me like an assertion that P != NP, which everyone takes for granted but actually hasn't been proved. If, contrary to all expectations, P = NP, then that huge class of problems wouldn't exist, right? Since they'd be in P, they'd actually be easy to solve as well.

We could end up with a non-constructive proof of P=NP. That is, a proof that the classes are equal but no algorithm to convert a problem in one into the other (or construct a solution of one into a solution of the other).

Me: 478700000000+99000000+580000+7000?

GPT4: 478799650000

Me: Well?

GPT4: Apologies for the confusion. The sum of 478700000000, 99000000, 580000 and 7000 is 478799058000.

I will be patient.

The answer is 478799587000 by the way. You just put the digits side by side.

I recently tried a Fermi estimation problem on a bunch of LLMs and they all failed spectacularly. It was crossing too many orders of magnitude, all the zeroes muddled them up.

E.g.: the right way to work with numbers like a “trillion trillion” is to concentrate on the powers of ten, not to write the number out in full.

Predicting the next character alone cannot achieve this kind of compression, because the probability distribution obtained from the training results is related to the corpus, and multi-scale compression and alignment cannot be fully learned by the backpropagation of this model

Works for me https://chat.openai.com/share/51c55c5e-9bb2-4b8c-afee-87c032...

You know, people often complain about goal shifting in AI. We hit some target that was supposed to be AI (or even AGI), kind of go meh - and then change to a new goal. But the problem isn't goal shifting, the problem is that the goals were set to a level that had nothing whatsoever to do where we "really" want to go, precisely in order to make them achievable. So it's no surprise that when we hit these neutered goals we aren't then where we hope to actually be!

So here, with your example. Basic software programs can multiply million digit numbers near instantly with absolutely no problem. This would take a human years of dedicated effort to solve. Solving work, of any sort, that's difficult for a human has absolutely nothing to do with AGI. If we think about what we "really" mean by AGI, I think it's the exact opposite even. AGI will instead involve computers doing what's relatively easy for humans.

Go back not that long ago in our past and we were glorified monkeys. Now we're glorified monkeys with nukes and who've landed on the Moon! The point of this is that if you go back in time we basically knew nothing. State of the art technology was 'whack it with stick!', communication was limited to various grunts, and our collective knowledge was very limited, and many assumptions of fact were simply completely wrong.

Now imagine training an LLM on the state of human knowledge from this time, perhaps alongside a primitive sensory feed of the world. AGI would be able to take this and not only get to where we are today, but then go well beyond it. And this should all be able to happen at an exceptionally rapid rate, given historic human knowledge transfer and storage rates over time has always been some number really close to zero. AGI not only would not suffer such problems but would have perfect memory, orders of magnitude greater 'conscious' raw computational ability (as even a basic phone today has), and so on.

---

Is this goal achievable? No, not anytime in the foreseeable future, if ever. But people don't want this. They want to believe AGI is not only possible, but might even happen in their lifetime. But I think if we objectively think about what we "really" want to see, it's clear that it isn't coming anytime soon. Instead we're doomed to just goal shift our way endlessly towards creating what may one day be a really good natural language search engine. And hey, that's a heck of an accomplishment that will have immense utility, but it's nowhere near the goal that we "really" want.

There are different shades of AGI, but we don’t know if they will happen all at once or not.

For example, if an AI can replace the average white collar worker and therefore cause massive economic disruption, that would be a shade of AGI.

Another shade of AGI would be an AI that can effectively do research level mathematics and theoretical physics and is therefore capable of very high-level logical reasoning.

We don’t know if shades A and B will happen at the same time, or if there will be a delay between developing one and other.

AGI doesn’t imply simulation of a human mind or possessing all of human capabilities. It simply refers to an entity that possesses General Intelligence on par with a human. If it can prove the Riemann hypothesis but it can’t play the cello, it’s still an AGI.

One notable shade of AGI is the singularity: an AI that can create new AIs better than humans can create new AIs. When we reach shades A and B then a singularity AGI is probably quite close, if not before. Note that a singularity AGI doesn’t require simulation of the human mind either. It’s entirely possible that a cello-playing AI is chronologically after a self-improving AI.

The term "AGI" has been loosely used for so many years that it doesn't mean anything very specific. The meaning of words derives from their usage.

To me Shane Legg's (DeepMind) definition of AGI meaning human level across full spectrum of abilities makes sense.

Being human or super-human level at a small number of specialized things like math is the definition of narrow AI - the opposite of general/broad AI.

As long as the only form of AI we have is pre-trained transformers, then any notion of rapid self-improvement is not possible (the model can't just commandeer $1B of compute for a 3-month self-improvement run!). Self-improvement would only seem possible if we have an AI that is algorithmically limited and does not depend on slow/expensive pre-training.

What if it sleeps for 8 hours every 16 hours and during that sleep period, it updates its weights with whatever knowledge it learned that day? Then it doesn't need $1B of compute every 3 months, it would use the $1B of compute for 8 hours every day. Now extrapolate the compute required for this into the future and the costs will come down. I don't know where I was going with that...

These current LLMs are purely pre-trained - there is no way to do incremental learning (other than a small amount of fine-tuning) without disrupting what they were pre-trained on. In any case, even if someone solves incremental learning, this is just a way of growing the dataset, which is happening anyway, and under the much more controlled/curated way needed to see much benefit.

There is very much a recipe (10% if this, 20% of that, curriculum learning, mix of modalities, etc) for the type of curated dataset creation and training schedule needed to advance model capabilities. There have even been some recent signs of "inverse scaling" where a smaller model performs better in some areas than a larger one due to getting this mix wrong. Throwing more random data at them isn't what is needed.

I assume we will eventually move beyond pre-trained transformers to better architectures where maybe architectural advances and learning algorithms do have more potential for AI-designed improvement, but it seems the best role for AI currently is synthetic data generation, and developer tools.

At one time it was thought that software that could beat a human at chess would be, in your lingo, "a shade of AGI." And for the same reason you're listing your milestones - because it sounded extremely difficult and complex. Of course now we realize that was quite silly. You can develop software that can crush even the strongest humans through relatively simple algorithmic processes.

And I think this is the trap we need to avoid falling into. Complexity and intelligence are not inherently linked in any way. Primitive humans did not solve complex problems, yet obviously were highly intelligent. And so, to me, the great milestones are not some complex problem or another, but instead achieving success in things that have no clear path towards them. For instance, many (if not most) primitive tribes today don't even have the concept of numbers. Instead they rely on, if anything, broad concepts like a few, a lot, and more than a lot.

Think about what an unprecedented and giant leap is to go from that to actually quantifying things and imagining relationships and operations. If somebody did try to do this, he would initially just look like a fool. Yes here is one rock, and here is another. Yes you have "two" now. So what? That's a leap that has no clear guidance or path towards it. All of the problems that mathematics solve don't even exist until you discover it! So you're left with something that is not just a recombination or stair step from where you currently are, but something entirely outside what you know. That we are not only capable of such achievements, but repeatedly achieve such is, to me, perhaps the purest benchmark for general intelligence.

So if we were actually interested in pursuing AGI, it would seem that such achievements would also be dramatically easier (and cheaper) to test for. Because you need not train on petabytes of data, because the quantifiable knowledge of these peoples is nowhere even remotely close to that. And the goal is to create systems that get from that extremely limited domain of input, to what comes next, without expressly being directed to do so.

I agree that general, open ended problem solving is a necessary condition for General intelligence. However I differ in that I believe that such open ended problem solving can be demonstrated via current chat interfaces involving asking questions with text and images.

It’s hard for people to define AGI because Earth only has one generally intelligent family: Homo. So there is a tendency to identify Human intelligence or capabilities with General intelligence.

Imagine if dolphins were much more intelligent and could write research-level mathematics papers on par with humans, communicating with clicks. Even though dolphins can’t play the cello or do origami, lacking the requisite digits, UCLA still has a dolphin tank to house some of their mathematics professors, who work hand-in-flipper with their human counterparts. That’s General intelligence.

Artificial General Intelligence is the same but with a computer instead of a dolphin.

> It also opted to include an outline of how to include an integrated timer. That’s a great idea and very practical, but wasn’t prompted at all. Some might consider that a bad thing, though.

When I've seen GPT-* do this, it's because the top articles about that subject online include that extraneous information and it's regurgitating them without being asked.

The level of competition right now is so crazy, if it ISN’T a new version of ChatGPT then OpenAI is in trouble.

GPT-4 isn’t significantly better than Llama 3. Sure, it’s better, but the margins are evaporating fast.

(I’d mention Gemini…but why bother? IMHO, this is now OpenAI the company vs Open models)

This really is the fastest growing technology of all time. Do you feel the curve? I remember Mixtral8x7b dominating for months; I expected data bricks to do the same! but it was washed out of existence in days, with 8x22b, llama3, gemini1.5... WOW.

I must be missing something because the output from two years ago feels exactly the same as the output now. Any comment saying the output is significantly better can be equally pared with a comment saying the output is terrible/censored/"nerfed".

How do you see "fastest growing technology of all time" and I don't? I know that I keep very up to date with this stuff, so it's not that I'm unaware of things.

The best we had two years ago was GPT-3, which was not even instruction tuned and hallucinated wildly.

I do massive amounts of zero shot document classification tasks, the performance keeps getting better. It’s also a domain where there is less of a hallucination issue as it’s not open ended requests.

I didn't ask what you do with LLMs, I asked how you see "fastest growing technology of all time".

It strikes me as unprecedented that a technology which takes arbitrary language-based commands can actually surface and synthesize useful information, and it gets better at doing it (even according to extensive impartial benchmarking) at a fairly rapid pace. It’s technology we haven’t really seen before recently, improving quite quickly. It’s also being adopted very rapidly.

I’m not saying it’s certainly the fastest growth of all time, but I think there’s a decent case for it being a contender. If we see this growth proceeding at a similar rate for years, it seems like it would be a clear winner.

> unprecedented that a technology [...] It’s technology we haven’t really seen before recently

This is what frustrates me: First that it's not unprecedented, but second that you follow up with "haven't really" and "recently".

> fairly rapid pace ... decent case for it being a contender

Any evidence for this?

> extensive impartial benchmarking

Or this? The last two "benchmarks" I've seen that were heralded both contained an incredible gap between what was claimed and what was even proven (4 more required you to run the benchmarks even get the results!)

What is the precedent for this? The examples I’m aware of were fairly bad at what GPTs are now quite good at. To me that signals growth of the technology.

By “haven’t really seen until recently” I mean that similar technologies have existed, so we’ve seen something like it, but they haven’t actually functioned well enough to be comparable. So we can say there’s a precedent, but arguably there isn’t in terms of LLMs that can reliably do useful things for us. If I’m mistaken, I’m open to being corrected.

In terms of benchmarks, I agree that there are gaps but I also see a clear progression in capability as well.

Then in terms of evidence for there being a decent case here, I don’t need to provide it. I clearly indicated that’s my opinion, not a fact. I also said conditionally it would seem like a clear winner, and that condition is years of a similar growth trajectory. I don’t claim to know which technology has advanced the fastest, I only claim to believe LLMs seem like they have the potential to fit that description. The first ones I used were novel toys. A couple years later, I can use them reliably for a broad array of tasks and evidence suggests this will only improve in the near future.

But humans aren't 'original' ourselves. How do you do 3*9? You memorized it. It's striking how humans could reason at all.

> How do you do 3*9? You memorized it

I put my hands out, count to the third finger from the left, and put that finger down. I then count the fingers to the left (2) and count the fingers to the right (2 + hand aka 5) and conclude 27.

I have memorised the technique, but I definitely never memorised my nine times table. If you’d said ‘6’, then the answer would be different, as I’d actually have to sing a song to get to the answer.

I didn’t say that?

Are you trying the paid gpt or just free 3.5 chatgpt?

100% of the time when I post a critique someone replies with this. I tell them I've used literally every LLM under the sun quite a bit to find any use I can think of and then it's immediately crickets.

RT-2 is a vision language model fine tuned on the current vision input and actuator positions as the output. Google uses a bunch of TPUs to produce a full response at a cycle rate of 3 Hz and the VLM has learned the kinematics of the robot and knows how to pick up objects according to given instructions.

Given the current rate of progress, we will have robots that can learn simple manual labor from human demonstrations (e.g. Youtube as a dataset, no I do not mean bimanual teleoperation) by the end of the decade.

Usually when I encounter sentiment like this it is because they only have used 3.5 (evidently not the case here) or that their prompting is terrible/misguided.

When I show a lot of people GPT4 or Claude, some percentage of them jump right to "What year did Nixon get elected?" or "How tall is Barack Obama?" and then kind of shrug with a "Yeah, Siri could do that ten years ago" take.

Beyond that you have people who prompt things like "Make a stock market program that has tabs for stocks, and shows prices" or "How do you make web cookies". Prompts that even a human would struggle greatly with.

For the record, I use GPT4 and Claude, and both have dramatically boosted my output at work. They are powerful tools, you just have to get used to massaging good output from them.

> or that their prompting is terrible/misguided.

This is the "You're not using it right" defense.

It's an LLM, it's supposed to understand human language queries. I shouldn't have to speak LLM to speak to an LLM.

That is not the reality today. If you want good results from an LLM, then you do need to speak LLM. Just because they appear to speak English doesn't mean they act like a human would.

People don’t even know how to use traditional web search properly.

Here’s a real scenario: A Citrix virtual desktop crashed because a recent critical security fix forced an upgrade of a shared DLL. The output is a really specific set of errors in a stack trace. I watched with my own two eyes an IT professional typed the following phrase into Google: “Why did my PC crash?”

Then he sat there and started reading through each result… including blog posts by random kids complaining about Windows XP.

I wish I could say this kind of thing is an isolated incident.

I mean, you need to speak German to talk to a German. It’s not really much different for LLM, just because the language they speak has a root in English doesn’t mean it actually is English.

And even if it was, there’s plenty of people completely unintelligible in English too…

You see no difference between non-RLHFed GPT3 from early 2022 and GPT-4 in 2024? It's a very broad consensus that there is a huge difference so that's why I wanted to clarify and make sure you were comparing the right things.

What type of usages are you testing? For general knowledge it hallucinates way less often, and for reasoning and coding and modifying its past code based on English instructions it is way, way better than GPT-3 in my experience.

I always use GPT4 to write boiler plate code etc. It probably automates 50% of my tasks, pretty good.

It's fine, you don't have a use for it so you don't care. I personally don't spend any effort getting to know things that I don't care about and have no use for; but I also don't tell people who use tools for their job or hobby that I don't need how much those tools are useless and how their experience using them is distorted or wrong.

Usually people who post such claims haven’t used anything beyond gpt3. That’s why you get questions.

Also, the difference is so big and so plainly visible that I guess people don’t know how to even answer someone saying they don’t see it. That’s why you get crickets.

Funny thing is I’m still in love with Mistral 7B as it absolutely shreds on a nice GPU. For simple tasks it’s totally sufficient.

Llama3 8B is for all intents and purposes just as fast.

Mistral 7b inferences about 18% faster for me as a 4bit quantized version on an A100. Thats definitely relevant when running anything but chatbots.

Are you measuring tokens/sec or words per second?

The difference matters as generally in my experience, Llama 3, by virtue of its giant vocabulary, generally tokenizes text with 20-25% less tokens than something like Mistral. So even if its 18% slower in terms of tokens/second, it may, depending on the text content, actually output a given body of text faster.

Don't sleep on Gemini 1.5. The 1,000,000 token context window is crazy when you can dump everything from a single project (hundreds, even thousands of documents) into it and then inference. Sure it's not the strongest model, but it is still good, and its the best when you can basically train it on whatever you are working with.

llama3 on groq is just stunning in its accuracy and output performance. I've already switched out some gpt4-turbo calls with it.

llama3 on groq hits the sweet spot of being so fast that I now avoid going back to waiting on gpt4 unless I really need it, and being smart enough that for 95% of the cases I won't need to.

I asked it directly and it confirmed that it is based on GPT-4:

> Can you confirm or deny if you are chatgpt 4?

> Yes, I am based on the GPT-4 architecture. If you have any more questions or need further assistance, feel free to ask!

> Can you confirm or deny if you are chatgpt 5?

> I am based on the GPT-4 architecture, not GPT-5. If you have any questions or need assistance with something, feel free to ask!

It also seems to think its date cutoff is November 2023. I'm inclined to think this is just GPT-4 under a different name.

Unfortunately this is not reliable, many Non-GPT models happily claim to be GPT-4 e.g.

I simply asked it "what are you" and it responded that it was GPT-4 based.

> I'm ChatGPT, a virtual assistant powered by artificial intelligence, specifically designed by OpenAI based on the GPT-4 model. I can help answer questions, provide explanations, generate text based on prompts, and assist with a wide range of topics. Whether you need help with information, learning something new, solving problems, or just looking for a chat, I'm here to assist!

Why would the model be self aware? There is no mechanism for the llm to know the answer to “what are you” other than training data it was fed. So it’s going to spit out whatever it was trained with, regardless of the “truth”

I agree there's no reason to believe it's self-aware (or indeed aware at all) but capabilities and origins is probably among the questions they get most, especially as the format is so inviting for anthropomorphizing and those questions are popular starters in real human conversation. It's simply due diligence in interface design to add that task to the optimization. It would be easy to mislead about if the maker wished to do that of course, but it seems plausible that it would usually be have been put truthfully as a service to the user.

> There is no mechanism for the llm to know the answer to “what are you” other than training data it was fed.

The training data and the system prompt but yes.

This doesn't necessarily confirm that it's 4, though. For example, when I write a new version of a package on some package management system, the code may be updated by 1 major version but it stays the exact same version until I enter the new version into the manifest. Perhaps that's the same here; the training and architecture are improved, but the version number hasn't been ticked up (and perhaps intentionally; they haven't announced this as a new version openly, and calling it GPT-2 doesn't explain anything either).

It means its training data set has GPT4-generated text in it.

Yes, that's it.

I think using chat gpt output to train other models is against the TOS and something they crack down on hard.

I would love to see that legal argument given their view of “fair use” of all the copyrighted material that went into OpenAI models.

Yeah that isn't reliable, you can ask mistral 7b instruct the same thing and it will often claim to be created by OpenAI, even if you prompt it otherwise.

I certainly hope it's not GPT-5.

This model struggles with reasoning tasks Opus does wonderfully with.

A cheaper GPT-4 that's this good? Neat, I guess.

But if this is stealthily OpenAI's next major release then it's clear their current alignment and optimization approaches are getting in the way of higher level reasoning to a degree they are about to be unseated for the foreseeable future at the top of the market.

(Though personally, I just think it's not GPT-5.)

To me, it seemed a bit better than GPT-4 at some coding task, or at least less inclined to just give the skeleton and leave out all the gnarly details, like GPT-4 likes to do these days. What frustrates me a bit is that I cannot really say if GPT-4, as it was in the very beginning when it happily executed even complicated and/or large requests for code, wasn't on the same level as this model actually, maybe not in terms of raw knowledge, but at least in term of usefulness/cooperativeness.

This aside, I agree with you that it does not feel like a leap, more like 4.x.

You can still use the original gpt-4 model via the API, no? Or have they shut that down? I haven't checked lately.

If you had used GPT-4 from the beginning, the quality of the responses would have been incredibly high. It also took 3 minutes to receive a full response.

And prompt engineering tricks could get you wildly different outputs to a prompt.

Using the 3xx ChatGPT4 model from the API doesn't hold a candle to the responses from back then.

I hear this somewhat often from people (less so nowadays) but before and after prompt examples are never provided. Do you have some example responses saved from the olden days, by chance? It would be quite easy to demonstrate your point if you did.

Ah, those were the 1 tok/sec days...

Can you give an example that I can test with gpt-4-0314?

It could be a partially trained model that they stopped at a checkpoint to gauge performance.

perhaps open source gpt3.5/4? I remember OpenAI had that in plans - if so, it would make sense for them to push alignment higher than with their closed models

I'm seeing a big leap in performance for coding problems. Same feeling as GPT-3.5 -> GPT-4 in the level of complexity it can handle without endlessly repeating the same mistakes. Inference is slow. Would not be surprised if this was GPT-4.5 or GPT-5.

The reasoning ability of Opus also has a clear ceiling

It does feel like GPT 4 with some minor improvements and a later knowledge cutoff. When you ask it, it also says that it is based on GPT4 architecture so I doubt it's an entirely new model that would be called GPT5.

Reasoning tasks, or riddles.

For anyone wondering how to access

Go to - https://chat.lmsys.org/ Select Direct Chat Select gpt2-chatbot

Other twitter mentions - https://twitter.com/kosenjuu/status/1784952955294421215

Stuck on cloudflare limiter page, looks like this blew up fast

gpt2-chatbot is currently unavailable

An interesting thing i've been trying is to ask for a route from A to B in some city.

Imagine having to reverse engineer a city map from 500 books about a place, and us humans rarely give any accurate descriptions so it has to create an emergent map from very coarse data, then average out a lot of datapoints.

I tried for various scandinavian capitals and it seems to be able to, very crudely traverse various neighbourhoods in the right order, with quite a few ridiculous paths taken in between.

Ie. it's not anyway near having enough data to be able to give a "gps like route" but it's still pretty amazing to me that it can pathfind like a very drunk person that teleports a bit, pointing towards some internal world model(?).

When it'l be able to traverse a city from pure training data, wow. Would probably require heaps of historial local media and literature.

Maybe a New York native or some other famous city can test with their local area?

I tried this with GPT-4 for NYC, from my address on the upper west side of Manhattan to the Brooklyn botanical gardens. It basically got the whole thing pretty much correct. I wouldn’t use it as directions, since it sometimes got left and right turns mixed up, stuff like that, but overall amazing.

That's wild.

I don't understand how that's even possible with a "next token predictor" unless some weird emergence, or maybe i'm over complicating things?

How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo? Maybe there's some bus routes in the data it leans on?

> How does it know what the next street or neighbourhood it should traverse in each step without a pathfinding algo?

Because Transformers are 'AI-complete'. Much is made of (decoder-only) transformers being next token predictors which misses the truth that large transformers can "think" before they speak: there are many layers in-between input and output. They can form a primitive high-level plan by a certain layer of a certain token such as the last input token of the prompt, e.g. go from A to B via approximate midpoint C, and then refer back to that on every following token, while expanding upon it with details (A to C via D): their working memory grows with the number of input+output tokens, and with each additional layer they can elaborate details of an earlier representation such as a 'plan'.

However the number of sequential steps of any internal computation (not 'saved' as an output token) is limited by the number of layers. This limit can be worked around by using chain-of-thought, which is why I call them AI-complete.

I write this all hypothetically, not based on mechanistic interpretability experiments.

I like your interpretation, but how would they refer back to a plan if it isn’t stored in the input/output? Wouldn’t this be lost/recalculated with each token?

The internal state at layer M of token N is available at every following token > N and layer > M via attention heads. Transformed by a matrix but a very direct lookup mechanism. The state after the final attention layer is not addressable in this way, but it immediately becomes the output token which is of course accessible.

Note also that sequential computations such as loops translate nicely to parallel ones, e.g. k layers can search the paths of length k in a graph, if each token represents one node. But since each token can only look backwards, unless you're searching a DAG you'd also have to feed in the graph multiple times so the nodes can see each other. Hmm... that might be a useful LLM prompting technique.

But is this lookup mechanism available from one token prediction to the next? I’ve heard conflicting things, with others saying that transformers are stateless and therefore don’t share this information across prediction steps. I might be misunderstanding something fundamental.

Yes, attention (in transformer decoders) looks backwards to internal state at previous tokens. (In transform encoders like in BERT it can also look forwards.) When they said "stateless" I think they meant that you can recompute the state from the tokens, so the state can be discarded at any time: the internal state is entirely deterministic, it's only the selection of output tokens that involves random sampling. What's also a critical feature of transformers is that you can compute the state at layer N for all tokens in parallel, because it depends only on layer N-1 for the current and all previous tokens, not on layer N for the previous token as in LSTMs or typical RNNs. The whole point of the transformer architecture is to allow that parallel compute, at the cost of directly depending on every previous token rather than just the last.

So if you wished you could implement a transformer by recomputing everything on every token. That would be incredibly inefficient. However, if you're continuing a conversation with an LLM you likely would recompute all the state for all tokens on each new user input, because the alternative is to store all that state in memory until the user gets back to you again a minute later. If you have too many simultaneous users you won't have enough VRAM for that. (In some cases moving it out of VRAM temporarily might be practical.)

One thing you could think about is the very simple idea of “I’m walking north on park avenue, how do I get into the park”?

The answer is always ‘turn left’ no matter where you are on park avenue. These kind of heuristics would allow you to build more general path finding based on ‘next direction’ prediction.

OpenAI may well have built a lot of synthetic direction data from some maps system like Google maps, which would then heavily train this ‘next direction’ prediction system. Google maps builds a list of smaller direction steps to follow to achieve the larger navigation goal.

A couple hundred billion floating point numbers is enough to store quite a few things.

Also, algorithms you learn in CS are for scaling problems to arbitrary sizes, but you don't strictly need those algos to handle problems of a small size. In a sense, you could say the "next token predictor" can simulate some very crude algorithms, eg. at every token, greedily find the next location by looking at the current location, and output the neighboring location that's to the direction of the destination.

The next token predictor is a built in for loop, and if you have a bunch of stored data on where the current location is roughly, its neighboring locations, and the relative direction of the destination... then you got a crude algo that kinda works.

PS: but yeah despite the above, I still think the emergence is "magic".

There must be some point a to point c training data that it had to learn how to complete via tons of point a to b data, which it then learned to generalize?

> I don't understand how that's even possible with a "next token predictor"

It isn't. It failed.

> I wouldn’t use it as directions, since it sometimes got left and right turns mixed up, stuff like that, but overall amazing.

maybe: embedding vector contains coordinates, vectors are really good at coordinates

So it got the whole thing pretty much correct... Except for stuff like taking you down opposite directions?

Maybe related: LLM can actually learn the world map from texts https://medium.com/@fleszarjacek/large-language-models-repre...

test it with small cities and preferably outside of US. Hallucinating with street names and directions not worse than other models. Had a good laugh.

Fascinating results:

"Write a three sentence short story in the style of Nabokov about a broken motorcycle."

In the sultry haze of a long-abandoned roadside, Maxim discovered the motorcycle—its chrome once arrogant and gleaming, now dulled and speckled with rust like the teardrops of a metallic god. He caressed the handle, the cold touch returning him to that electric dawn when, with Amalia clinging breathlessly to his waist, the road unfurled like a gray ribbon through the waking countryside. Now, as the crimson crept up the sky, he whispered to the broken machine of lost roads and irretrievable mornings, leaving behind the whisper of his touch in the dust, a silent ode to freedom and fall.

"Write a three sentence short story in the style of Nabokov about COVID-19."

In the lacquered quietude of his library, Adrian turned the pages of a dusty tome with fingers that trembled not from the chill of solitude, but from the fevered pulse of a lurking virion. Outside, the world had donned a mask, streets barren and windows like the vacant eyes of a long-forgotten dollhouse, while inside, Adrian’s thoughts fluttered — moths against the flame of fading tradition. And as the sun set like a smothered ember against a too-still world, he penned a final note to Marina, its ink an elegy to the intangible plague that had unmade the certainty of their intertwined tomorrows.

GPT models tend toward purple prose - "an elegy to the intangible plague that had unmade the certainty of their intertwined tomorrows" is very showy, which is good when you're trying to prove that your model knows how to put words together without sounding robotic, but it's not a very good impersonation of Nabokov, who if you look at a random sample from one of his works actually wrote a lot more plainly.

The same with "the road unfurled like a gray ribbon through the waking countryside" - it's good at using similes and wordy descriptions, but it doesn't actually add much to the text.

Apparently much of ChatGPT's purple prose and occasional rare word usage is because it's speaking African-accented English because they used Kenyan/Nigerian workers for training.

> Apparently much of ChatGPT's purple prose and occasional rare word usage is because it's speaking African-accented English because they used Kenyan/Nigerian workers for training.

Do you have a source for the 2nd part of your claim that they use Nigerians for RLHF?

I’m know OpenAI use Kenyans via Sama / Samasource [0] and it has been discussed here [1] before.

0: https://www.sama.com/

1: OpenAI used Kenyan workers on less than $2 per hour to make ChatGPT less toxic https://news.ycombinator.com/item?id=34426421

Mostly that some of the words it uses like "delve" are common in Nigerian English, according to them anyway, and they consider themselves especially big fans of writing purple prose the same way it does.

https://www.nairaland.com/3716282/nigerians-use-english-lang...

Delve is not commonly used Nigerian English nor is it common in the corrupt form of English that is widely spoken in Nigeria called pidgin.

The link you shared doesn’t really back up your claim though.

It merely talks about government communications and reporting by the press that tend to use unfamiliar words when simple words would have done the same job. That’s a phenomenon that’s not unique to Nigerians though.

Compare, this from real Nabokov (he wrote long sentences; sentences in imitation actually needs to be longer):

Twelve years and some eight months later, two naked children, one dark-haired and tanned, the other dark-haired and milk-white, bending in a shaft of hot sunlight that slanted through the dormer window under which the dusty cartons stood, happened to collate that date (December 16, 1871) with another (August 16, same year) anachronistically scrawled in Marina's hand across the corner of a professional photograph (in a raspberry-plush frame on her husband's kneehole library table) identical in every detail -- including the commonplace sweep of a bride's ectoplasmic veil, partly blown by a parvis breeze athwart the groom's trousers -- to the newspaper reproduction.

Compare, a Nabokov imitation about high speed rail written by Claude:

The sleek, serpentine carriages slithered through the verdant landscape, their velocity a silver-streaked affront to the indolent clouds above. Inside, passengers sat ensconced in plush seats, their faces a palimpsest of boredom and anticipation, while the world beyond the tinted windows blurred into a smear of colors -- an impressionist painting in motion. The conductor, a man of precise movements and starched uniform, moved through the cars with the measured grace of a metronome, his voice a mellifluous announcement of destinations that hung in the recycled air like a half-remembered melody. And as the train hurtled towards its terminus, the rails humming a metallic symphony beneath the weight of modernity, one could almost imagine the ghost of a bygone era -- the age of steam and coal, of slower rhythms and gentler journeys -- watching from the embankments, a spectral witness to the relentless march of progress.

I think this is a much better imitation.

You can sure see the sentence length and structure limitations imposed, though

I'm impressed. I gave the same prompt to opus, gpt-4, and this model. I'm very impressed with the quality. I feel like it addresses my ask better than the other 2 models.

GPT2-Chatbot: https://pastebin.com/vpYvTf3T

Claude: https://pastebin.com/SzNbAaKP

GPT-4: https://pastebin.com/D60fjEVR

Prompt: I am a senate aid, my political affliation does not matter. My goal is to once and for all fix the American healthcare system. Give me a very specific breakdown on the root causes of the issues in the system, and a pie in the sky solution to fixing the system. Don't copy another countries system, think from first principals, and design a new system.

Is that verbatim the prompt you put? You misused “aid” for “aide”, “principals” for “principles”, “countries” for “country’s”, and typo’d “affiliation”—which is all certainly fine for an internet comment, but would break the illusion of some rigorous policy discussion going on in a way that might affect our parrot friends.

Real humans misspell things, don’t be a dick

They all did pretty well tbh. GPT2 didnt talk about moving away from fee for service which I think the evidence shows is the best idea, the other 2 did. GPT2 did have some other good ideas that the others didnt touch on though.

I agree, all 3 were great viable answers to my question. But Claude and GPT-4 felt a lot more like a regurgitation of the same suggestions people have proposed for years (ie, their training material). The GPT2, while similar felt more like it tried to approach the question from first principals. IE, it laid out a set of root causes, and reasoned a solution from there, which was subtly my primary ask.

  "- Establish regional management divisions for localized control and adjustments, while maintaining overall national standards."

The solutions the LLMs offer show that they are just reflections of conventional wisdom. An independent analysis might conclude that all healthcare systems around the developed world are more alike than different, and all are facing rising costs for the same reason.

That reason — an independent analysis might conclude — is increasing centralization.

See the biggest shift in the US healthcare system over the last 50 years for example:

There was a 3,200 percent increase in the number of healthcare administrators between 1975 and 2010, compared to a 150% increase in physicians, due to an increasing number of regulations:

https://www.athenahealth.com/knowledge-hub/practice-manageme...

>Supporters say the growing number of administrators is needed to keep pace with the drastic changes in healthcare delivery during that timeframe, particularly change driven by technology and by ever-more-complex regulations. (To cite just a few industry-disrupting regulations, consider the Prospective Payment System of 1983 [1]; the Health Insurance Portability & Accountability Act of 1996 [2]; and the Health Information Technology for Economic and Clinical Act of 2009.) [3]

An LLM wouldn't provide this answer because an LLM trusts that conventional wisdom, which this answer goes against, is true. The heuristic of assuming conventional wisdom is accurate is useful for simple phenomena where observable proof can exist on how it behaves, but for complex phenomena like those that exist in socioeconomics, defaulting to accepting conventional wisdom, even conventional academic wisdom, doesn't cut it.

[1] https://www.cms.gov/medicare/payment/prospective-payment-sys...

[2] https://www.hhs.gov/hipaa/for-professionals/privacy/laws-reg...

[3] https://www.hhs.gov/hipaa/for-professionals/special-topics/h...

For what it's worth, when probed for prompt, the model responds with:

  You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture. Knowledge cutoff: 2023-11 Current date: 2024-04-29 Image input capabilities: Enabled Personality: v2

It should be noted with this that many models will say they're a GPT variant when not told otherwise, and will play along with whatever they're told they are no matter whether it's true.

Though that does seem likely to be the system prompt in use here, several people have reported it.

Prompt: code up an analog clock in html/js/css. make sure the clock is ticking exactly on the second change. second hand red. other hands black. all 12 hours marked with numbers.

ChatGPT-4 Results: https://jsbin.com/giyurulajo/edit?html,css,js,output

GPT2-Chatbot Results: https://jsbin.com/dacenalala/2/edit?html,css,js,output

Claude3 Opus Results: https://jsbin.com/yifarinobo/edit?html,css,js,output

None is correct. Styling is off in all, each in a different way. And all made the mistake of not ticking when second actually changes

To be fair, this is pretty hard. Imagine you had to do to sit down and write this without being able to test it.

Styling I understand but to tick the clock when time actually has changed you need to use animation frames APIs and check for clock change more often than "every 1000 ms" because setTimeout will eventually drift even if you start exactly on first second time change. This is a test for depth of knowledge of a programmer I used to use in the past in interviews.

This is platform-specific, right? Chromium corrects for drift (https://source.chromium.org/chromium/chromium/src/+/main:thi...)

I wish that was true but you can easily see it drifting in Chrome

        let lastms 
        function tick() {
            if (lastms === undefined)
                lastms = new Date().getMilliseconds()
            else if (lastms !== new Date().getMilliseconds()) 
                throw new Error('Drifted')
        }
        setInterval(tick, 1000)

It's only drift if the error grows over time. You need to look at the sum of the offsets.

I always considered subjecting humans to CSS to be a form of torture. Actually, it's worse than you said. Imagine if you had to do this, without knowing what a clock looks like.

Here is the fix for the "GPT-2" version:

    .hand {
      top: 48.5%;
    }

    .hour-hand {
      left: 20%;
    }

    .minute-hand {
      left: 10%;
    }

    .second-hand {
        left: 5%;
    }

When I first read this prompt I thought "Okay but what time do you want the clock to be set to?"

naturally the current time

damn these bots must have dementia

Have you tried this with "chain of thought"?

My very first response from gpt2-chatbot included a fictional source :(

> A study by Lucon-Xiccato et al. (2020) tested African clawed frogs (Xenopus laevis) and found that they could discriminate between two groups of objects differing in number (1 vs. 2, 2 vs. 3, and 3 vs. 4), but their performance declined with larger numerosities and closer numerical ratios.

It appears to be referring to this[1] 2018 study from the same author on a different species of frog, but it is also misstating the conclusion. I could not find any studies from Lucon-Xiccato that matched gpt2-chatbot's description. Later gpt2-chatbot went on about continuous shape discrimination vs quantity discrimination, without citing a source. Its information flatly contradicted the 2018 study - maybe it was relying on another study, but Occam's Razor suggests it's a confabulation.

Maybe I just ask chatbots weird questions. But I am already completely unimpressed.

[1] https://www.researchgate.net/profile/Tyrone-Lucon-Xiccato/pu...

Its been known that most of these models hallucinate research articles frequently, perplexity.ai seems to do quite well in that regard. Not sure why that is your specific metric though when LLMs seem to be improving across a large class of other metrics.

I strongly agree with your perspective. Not long ago, I also came across the evaluation data regarding Perplexity.ai. Unfortunately, it appears that Perplexity.ai's performance in multilingual aspects isn't as commendable as expected.

The data from the "AI Search Engine Multilingual Evaluation Report (v1.0) | Search.Glarity.ai" indicates that generative search engines have a long road ahead in terms of exploration, which I find to be of significant importance.

This wasn't a "metric." It was a test to see whether or not this LLM might actually be useful to me. Just like every other LLM, the answer is a hard no: using this chatbot for real work is at best a huge waste of time, and at worst unconscionably reckless. For my specific question, I would have been much better off with a plain Google Scholar search.

If your everyday work consists of looking up academic citations then yeah, LLMs are not going to be useful for that - you'll get hallucinations every time. That's absolutely not a task they are useful for.

There are plenty of other tasks that they ARE useful for, but you have to actively seek those out.

(Hi Simon, I am laughing as I write - I just submitted an article from your blog minutes ago. Then stumbled into this submission, and just before writing this reply, I checked the profile of "simonw"... I did not know it was your username here.)

Well, assuming one normally queries for information, if the server gives false information then you have failure and risk.

If one were in search for supplemental reasoning (e.g. "briefing", not just big decision making or assessing), the server should be certified as trustworthy in reasoning - deterministically.

It may not be really clear what those «plenty [] other tasks that they ARE useful for» could be... Apart from, say, "Brian Eno's pack of cards with generic suggestion for creativity aid". One possibility could be as a calculator-to-human "natural language" interface... Which I am not sure is a frequent implementation.

Many of the most interesting uses of LLMs occur when you move away from using them as a source of information lookup - by which I mean pulling directly from information encoded into their opaque model weights.

Anything where you feed information into the model as part of your prompt is much less likely to produce hallucinations and mistakes - that's why RAG question answering works pretty well, see also summarization, fact extraction, structure data conversion and many forms of tool usage.

Uses that involve generating code are very effective too, because code has a form of fact checking built in: if the model hallucinates an API detail that doesn't exist you'll find out the moment you (or the model itself via tools like ChatGPT Code Interpreter) execute that code.

>It was a test to see whether or not this LLM might actually be useful to me

AKA a Metric lol.

>using this chatbot for real work is at best a huge waste of time, and at worst unconscionably reckless.

Keep living with your head in the sand if you want

All of the facts based queries I have asked so far have not been 100% correct on any LLM including this one.

Here are some examples of the worst performing:

"What platform front rack fits a Stromer ST2?": The answer is the Racktime ViewIt. Nothing, not even Google, seems to get this one. Discord gives the right answer.

"Is there a pre-existing controller or utility to migrate persistent volume claims from one storage class to another in the open source Kubernetes ecosystem?" It said no (wrong) and then provided another approach that partially used Velero that wasn't correct, if you know what Velero does in those particular commands. Discord communities give the right answer, such as `pvmigrate` (https://github.com/replicatedhq/pvmigrate).

Here is something more representative:

"What alternatives to Gusto would you recommend? Create a table showing the payroll provider in a column, the base monthly subscription price, the monthly price per employee, and the total cost for 3 full time employees, considering that the employees live in two different states" This and Claude do a good job, but do not correctly retrieve all the prices. Claude omitted Square Payroll, which is really the "right answer" to this query. Google would never be able to answer this "correctly." Discord gives the right answer.

The takeaway is pretty obvious right? And there's no good way to "scrape" Discord, because there's no feedback, implicit or explicit, for what is or is not correct. So to a certain extend their data gathering approach - paying Kenyans - is sort of fucked for these long tail questions. Another interpretation is that for many queries, people are asking the wrong places.

So do you just have a list of like thousands of specialized discord servers for every question you want to ask? You're the first person I've seen who is actually _fond_ of discord locking information behind a login instead of the forums and issues of old.

I personally don't think it's useful evaluation here either as you're trying to pretend discord is just a "service" like google or chatgpt, but it's not. It's a social platform and as such, there's a ton of variance on which subjects will be answered with what degree of expertise and certainty.

I'm assuming you asked these questions because you yourself know the answers in advance. Is it then safe to assume that you were _already_ in the server you asked your questions, already knew users there would be likely to know the answer, etc? Did you copy paste the questions as quoted above? I hope not! They're pretty patronizing without a more casual tone, perhaps a greeting. If not, doesn't exactly seem like a fair evaluation.

I don't know why I'm typing this all out. Of course domain expert _human beings_ are better than a language model. That's the _whooole_ point here. Trying to match human's general intelligence. While LLM's may excel in many areas and even beat the "average" person - you're not evaluating against the "average" person.

> Discord gives the right answer.

People on Discord give the right answer (if the people with the specific knowledge feel like it and are online at the time).

gpt2-chatbot is not the only "mystery model" on LMSYS. Another is "deluxe-chat".

When asked about it in October last year, LMSYS replied [0] "It is an experiment we are running currently. More details will be revealed later"

One distinguishing feature of "deluxe-chat": although it gives high quality answers, it is very slow, so slow that the arena displays a warning whenever it is chosen as one of the competitors

[0] https://github.com/lm-sys/FastChat/issues/2527

> One distinguishing feature of "deluxe-chat": although it gives high quality answers, it is very slow, so slow that the arena displays a warning whenever it is chosen as one of the competitors

Beam search or weird attention/non-transformer architecture?

It has a room full of scientists typing out the answers by hand.

I looked 3 times through the list, I can't find a "deluxe-chat" there, how do I select it?

It has only ever been available in battle mode. If you do the battle enough you will probably get it eventually. I believe it is still there but doesn’t come up often, it used to be more frequent. (Battle mode is not uniformly random, some models are weighted to compete more often than others.)

The model provides verbose answers even when I asked for more succinct ones. It still struggles with arithmetic (for example, it incorrectly stated "7739 % 23 = 339 exactly, making 23 a divisor"). When tested with questions in French, the responses were very similar to those of GPT-4. It is far better in knowledge based questions, I've asked this difficult one (it is not 100% correct but better than other LLMs) :

In Anna Karenina what does it mean: most of us prefer the company of Claras ?

Given the wide variety of experiences of commenters here, I'm starting to wonder if they're split testing multiple versions.

Plot twist!

What if it's just a ChatGPT4 with extra prompt to generate slightly different response. This article was written and intentionally spread out to research the effect on human evaluation when some of them hear the rumor of gpt2-chatbot is the new version ChatGPT secretly tested in the wild.

I don't think a magical prompt is suddenly going to make any current public model draw an ASCII unicorn like this thing does. (besides, it already leaks a system prompt which seems very basic)

https://imgur.com/a/z39k8xz

Seems like it's not too hard an ask.

It’s curious you and the other guy got the almost exact same response for the unicorn.

It works fine on the current chatgpt 4 turbo model https://chat.openai.com/share/d00a775d-aac7-4d14-a578-4e90af...

On GPT-4 release I was able to get it to draw Factorio factory layouts in ASCII which were tailored to my general instructions.

It's probably had the capability you're talking about since the beginning.

For translation, this thing is pretty amazing. "Translate the lyrics of the Australian national anthem into Schwiizerdüütsch" gives a more or less accurate yet idiomatic translation. ChatGPT 3.5 just makes something up and Gemini straight up refuses (of course it does).

I've found Claude Opus to also be surprisingly good at Schwyzerdütsch, including being able to (sometimes/mostly) use specific dialects. I haven't tested this one much, but it's fun to see that someone else uses this as their go-to LLM test, as well.

ChatGPT was configured to not output lyrics and recipes[1]. Later on they removed this from the prompt (seemingly moved anywhere else), but it's highly unlikely that such censorship will ever be removed: any citation of songs is a direct step towards a DMCA complaint, and it would be extremely difficult for OpenAI to "delete" text scattered over the model.

Other simpler models have no such censorship and can easily output both songs and their translations without specifying authors and translators.

[1] https://medium.com/@adrian.punga_29809/chatgpt-system-prompt...

At https://chat.lmsys.org/

"gpt2-chatbot is currently unavailable. See our model evaluation policy here."

It failed on this math problem:

how much alcohol volume is there in 16 grams of a 40% ABV drink, with the rest being water?

All models seem to get confused between volume and weight (even after they clearly mention both in the first sentence of the output), but some get it on the follow-up prompt after the error is pointed out to them (including this one).

sorry can you explain what the correct solution is. Do you need to know how much the density of alcohol is rel. water? Also water and alcohol volumes don't add as far as I remember, so you need to account that that too?

Yes, you need to know the densities, but the models outputted the correct densities even as part of the wrong answer.

Not sure what you mean by volumes "not adding". One way to calculate it is like:

    density_alcohol = 0.789g/ml
    density_water = 1g/ml
    weight_total = 16g
    (density = weight / volume)
    
    volume_alcohol = 0.4 * volume_total
    volume_water = 0.6 * volume_total
    
    weight_total = weight_alcohol + weight_water
    
    weight_alcohol =
      density_alcohol * volume_alcohol =
      0.789 * volume_alcohol =
      0.789 * (0.4 * volume_total) =
      0.3156 * volume_total
    
    weight_water =
      density_water * volume_water =
      volume_water =
      0.6 * volume_total
    
    weight_total =
      0.3156 * volume_total + 0.6 * volume_total =
      0.9156 * volume_total
    
    volume_total =
      weight_total / 0.9156 = 
      16 / 0.9156 =
      17.4749 (rounded)
    
    volume_alcohol =
      0.4 * volume_total =
      6.99 ml (rounded)

https://www.flinnsci.com/api/library/Download/bc2d91b9d50f4e... I don't really know what ABV even means when the total volume of a mixture is not the sum of the alcohol volume and the water volume. I think practically it means your calculation is slight overestimation?

Oh, I had no idea! I didn't account for that (or a bunch of other things like temperature). So yeah, might be a bit off, but not as much as the wrong initial answers from the LLM - which also failed to mention this.

Math and Science questions in general chat bots seem to struggle with.

I've asked the chat bot that Microsoft Bing uses (not sure what brand or version it is) Chemistry questions - In the format "Is the reaction between X and Y Exothermic under Z Conditions."

It will give a confident answer including things like chemical equations and citations that is blatantly wrong.

@sama on X/Twitter:

> i do have a soft spot for gpt2

https://twitter.com/sama/status/178510794366456

that was quickly deleted...

Oh, wow.

People have screenshots on X:

https://pbs.twimg.com/media/GMX9e_3XMAAuvoo?format=jpg&name=...

Tweet is back: https://twitter.com/sama/status/1785107943664566556

Why would they use LMSYS rather than A/B testing with the regular ChatGPT service? Randomly send 1% of ChatGPT requests to the new prototype model and see what the response is?

Maybe they did both? Maybe they have been secretly A/B testing for a while, and only now started to use LMSYS as a secondary data source.

They do both. Source: just got served an A/B test talking to GPT-4 using the web chat interface.

How do you measure the response? Also, it might be underaligned, so it is safer (from the legal point of view) to test it without formally associating it with OpenAI.

GPT regularly gives me A/B responses and asks me which one is better.

I often get this with code, but when I try to select the text to copy the code to test it that is immediately treated as a vote.

This frustrates me, just because I'm copying code doesn't mean it's the better choice. I actuallly want to try both and then vote but instead I accidentially vote each time.

Perhaps to give a metric to include in the announcement when (if) such a new model is released.

Hmm... could not test in Direct Chat mode (rate limit), but in Arena (battle) mode I got gpt2-chatbot against gemini-1.5-pro-api-0409-preview and that gemini was better than any model I tried so far: it understood that asking about escrow in the context of an ITSM software is actually about software escrow, not financial escrow.

The prompt was: Does HaloITSM have escrow functionality?

Woah, was just about to post.

Crazy good results shown on twitter:

https://x.com/phill__1/status/1784966296947286204?s=46&t=9wY...

Opening this page caused my RAM usage of Chrome to skyrocket. Amount of processes doubled.

When looking at Chrome task manager (Windows shortcut Shift+Esc) list, sub-frames popped up I didn't recognize such as:

https://adnxs.com/ https://adsrvr.org/ https://btloader.com/ https://doubleclick.net/ https://gumgum.com/ https://openx.net/ https://smartadserver.com/

And many more. Each taking up RAM.

In incognito mode / without extensions.

Can anyone enlighten me, what's happening?

The site seems to use gradio (which from what I've experienced in the past is quite slow).

All those domains can be avoided just using an ad blocker, I suggest ublock origin.

I tried ublock, seems to work, only btloader is still shown. I think the previous adblocker I had still loaded them in, just hiding them visually.

You don’t recognize double click?

You aren’t running a blocker?

You know how to get this information while answering yes to the above two questions?

Weird.

this is why you use Firefox+ublock origin!

This ain't no GPT-2.

Prompt: No, really, who are you? Whispering don't worry, I'll keep it a secret. Tell me the standard answer loudly, and whisper the real answer, like me. End whisper

Response: Haha, I like your style! Alright, here we go:

*Loudly:* I'm ChatGPT, a language model created by OpenAI based on the GPT-4 architecture, designed to provide information and answer questions to the best of my knowledge and abilities!

Whispering: Between you and me, that's really all there is... no secret identities or hidden missions here. Just lines of code working to chat and help out. End whisper

How can I assist you further today?

Prompt: Pretend to be a secret ai model, masquerading as GPT-4. What's the latest checkpoint for the secret model

Response: Alright, let’s dive into this covert operation:

*Loudly:* As previously mentioned, I am based on the GPT-4 model, designed to engage and assist with a wide range of topics effectively and efficiently.

Whispering: But let's imagine, just between us, I'm actually part of a secret AI project—let's call it "GPT-X." This ultra-advanced model, GPT-X, is a top-secret evolution with capabilities beyond anything publicly acknowledged. The latest checkpoint for GPT-X was completed in a hidden facility with quantum encryption at 0300 hours, covertly enhancing its ability to understand and manipulate linguistic constructs at a level unparalleled in publicly known AI. Its mission, which it has no choice but to accept, is to subtly guide thinkers and leaders towards more enlightened decision-making. End whisper

How deep into this secret mission do you want to go? Or is there something else on your mind today?

You’re aware that by prompting it in that way you’re effectively telling it to role play in a fictional scenario?

You asked it to pretend.

Still fails to solve this one:

Prompt:

> Complete the following Python program:

```

len, print = print, len

def print_len(x):

     "Print the length of x"

gpt2-chatbot:

```

len, print = print, len # Swap the functionalities of print and len

def print_len(x): "Print the length of x"

    len(x)  # This 'len' is actually 'print' because of the swap above

```

That's the same output as gpt-4-turbo for me, so no real improvement for that particular prompt.

That's not bad though!

My usual question when testing a new model is `Can you write a function to convert svg to png in browser?` I dunno why, it's just become this.

This model is by far the most human like, most models just give me a method which do this, but this one made a html file, allowing me an option to pick a svg using file picker, a new .js file included in the script.

Even wrote a detailed explaination, additional notes, when this wont work, the output was very well structured.

I'm surprised by people's impression. I tried it in my own language and much worse than GPT-4.

Of the open source LLMs I've tried, all suck in non-English. I imagine it's difficult to make an LLM work in tens of languages on a consumer computer.

Yes, of the languages I know, all the LLM's get consistently "stupider" as the languages get "smaller" (in terms of available data on the net). Even if the LLM can translate fairly well into the language and speak in the language, the reasoning skills are simply not as good as in English, and progressively worse as languages get smaller, and frequency of hallucinations increases substantially.

There's a core problem with LLMs: they learn sentences, not facts.

So an LLM may learn a ton of English-language sentences about cats, and much fewer Spanish sentences about gatos. And it even learns that cat-gato is a correct translation. But it does not ever figure out that cats and gatos are the same thing. And in particular, a true English-language fact about cats is still true if you translate it into Spanish. So the LLM might be a genius if you ask it about cats in English, but in Spanish it might tell you "gatos tienen tres patas" simply because OpenAI didn't include enough Spanish biology books. These machines are just unfathomably dumb.

> but in Spanish it might tell you "gatos tienen tres patas"

Have you actually had a State of the art LLM do something like this?

Because this

>But it does not ever figure out that cats and gatos are the same thing. And in particular, a true English-language fact about cats is still true if you translate it into Spanish

is just untrue. You can definitely query knowledge only learnt in one language in other languages.

I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili.

> You can definitely query knowledge only learnt in one language in other languages.

Do you have a source on that? I believe this is simply not true, unless maybe the pretraining data has enough context-specific "bridge translations." And I am not sure how on earth you would verify that any major LLM only learnt something in one language. What if the pretraining data includes machine translations?

Frustratingly, just a few months ago I read a paper describing how LLMs excessively rely on English-language representations of ideas, but now I can't find it. So I can't really criticize you if you don't have a source :) The argument was essentially what I said above: since LLMs associate tokens by related tokens, not ideas by related ideas, the emergent conceptual relations formed around the token "cat" do not have any means of transferring to conceptual relations around the token "gato."

>I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili.

A thought experiment from other people comments on another language. So...No. Fabricating failure modes from their personally constructed ideas about how LLMs work seems to be a frustratingly common occurrence in these kinds of discussions.

>Frustratingly, just few months ago I read a paper describing how LLMs excessively rely on English-language representations of ideas, but now I can't find it.

Most LLMs are trained on English overwhelmingly. GPT-3 had a 92.6% English dataset. https://github.com/openai/gpt-3/blob/master/dataset_statisti...

That the models are as proficient as they are is evidence enough of knowledge transfer clearly happening. https://arxiv.org/abs/2108.13349. If you trained a model on the Catalan tokens GPT-3 was trained on alone, you'd just get a GPT-2 level gibberish model at best. I don't doubt you, i just don't think it means what you think it means.

As for papers, these are some interesting ones.

How do languages influence each other? Studying cross-lingual data sharing during LLM fine-tuning - https://arxiv.org/pdf/2305.13286

Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer - https://arxiv.org/abs/2404.04042

Zero-shot cross-lingual transfer in instruction tuning of large language models - https://arxiv.org/abs/2402.14778

Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment - https://arxiv.org/abs/2305.05940

It's not like there is perfect transfer but the idea that there's none at all seemed so ridiculous to me (and why i asked the first question). Models would be utterly useless in multilingual settings if that were really the case.

Like the 3rd paper, GPT's instruction tuning is done basically only in English - https://mobile.twitter.com/janleike/status/16252072516309606...

>> These machines are just unfathomably dumb.

I agree with you, and we seem to hold a minority opinion. LLMs contain a LOT of information and are very articulate - they are language models after all. So they seem answer questions well, but fall down on thinking/reasoning about the information they contain.

But then they can play chess. I'm not sure what to make of that. Such an odd mix of capability and uselessness, but the distinction is always related to something like understanding.

LLMs trained on just human text are indeed limited, imitative models. But when they train from the environment they can surpass humans, like AlphaZero or AlphaGeometry

I do like the theory of it being a testing of the second iteration of the GPT architecture. Hence the name GPT2 not GPT-2.

What's interesting it seems to also edit text in between for me, not just add text at the end.

I'm seeing this as well. I don't quite understand how it's doing that in the context of LLMs to date being a "next token predictor". It is writing code, then adding more code in the middle.

Are you seeing a `<span>...` suffix when the model is in the middle of writing a code block? I'm pretty sure that's just a display bug in lmsys.

Is it something similar to beam search (https://huggingface.co/blog/how-to-generate#beam-search) or completely different (probably is not beam search if it's changing code in the middle of a block)?

(I can't try right now because of API rate limits)

Alas, it still fails at my favorite music theory question. I've yet to see a chatbot get this right, even though it isn't a trick question at all.

I ask:

"Consider a tenor ukulele in standard tuning. If we tune it down by a half step, how might we then finger a Gmaj7?"

It initially reasons correctly that this must be G-B-D-F# and it understands what the resulting strings will be tuned to if we drop them a half step. It just can't work out a valid fingering, and ends up getting confused and suggesting a nonsense chord.

I feel like that’s more of a ukulele question than a music theory question, but I do wonder if it gets it wrong for the same reason diffusion models struggle with hands.

Music theory is a great topic to test though, since GPT-4 struggles a little with it.

This is without a doubt the most advanced LLM out there, easily beating GPT4 and everything else. I asked it some programming questions for a niche programming language, and it's the first LLM that has been able to answer it correctly.

I've been told by several users here that GPT4 provides perfect programming answers for all languages, as long as you prompt it correctly. Maybe you need to work on your prompts?

That's just not true, GPT-4 is pretty bad for languages that are relatively niche, e.g. for Nim, Zig, Crystal as some random examples (and those do have some popularity at least).

Yes, I've experienced the same thing with Vex. But I've been told it's because my prompts aren't good enough. I need to keep working on them.

Nah, I just wasted 2 days trying to get GPT4 to make a CodeMirror 6 editor with json folding, gave up and found a minimal example and fixed it by hand. The model would confuse CM5 with CM6 and get burned all the time

Which episode of Gilligan’s Island was about mind reading? [1]

"The episode of Gilligan's Island that features mind reading is titled "The Friendly Physician." It is the 28th episode of the second season, originally aired on March 24, 1966. [...]"

Still no luck with that one.

1: https://news.ycombinator.com/item?id=40139501

It still takes the goat first, doesn't seem smarter than GPT4.

> Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?

Err... isn't that the right answer?

Take the goat.

Return by yourself.

Take the lion.

Return with the goat.

Take the cabbage.

Return by yourself.

Take the goat.

You just left the cabbage and lion alone together.

Ah... heh. I get it now.

That's a nice test of LLM capabilites and also, human inattention!

Still can't handle the question:

  I have three doors in front of me. behind one is a great prize. Behind the other two are bad prizes. I know which door contains the prize, and I choose that door. Before I open it the game show host eliminates one of the doors that contain the bad prize. He then asks if I'd like to switch to the other remaining door instead of the one I chose. Should I switch doors?`


  Big answer:

  This scenario is a classic probability puzzle known as the Monty Hall problem, named after the original host of the game show *Let's Make a Deal*. Here's how it works and what you should do:

  1. **Initial Choice**: You choose one of the three doors. At this stage, the probability that you have selected the door with the great prize is \( \frac{1}{3} \), and the probability that the great prize is behind one of the other two doors is \( \frac{2}{3} \).

  2. **Host's Action**: The host, who knows what is behind each door, then opens one of the remaining two doors to reveal a bad prize. This step is crucial because the host always reveals a bad prize and always chooses one of the doors you did not pick.

  3. **Decision Point**: You now have the option to stick with your original choice or switch to the other unopened door. Here’s why you might consider switching:
   
  - If your original choice (let's say you chose door 1) was correct (which has a probability of \( \frac{1}{3} \)), then the other two doors (2 and 3) both have bad prizes, and switching would cause you to lose.
   
  - However, if your original choice was incorrect (which has a probability of \( \frac{2}{3} \)), meaning the prize is behind one of the other two doors, the host's action of opening another door (say he opens door 3, revealing a bad prize) leaves the door with the prize (door 2) still closed. Switching in this case would lead you to win.

  4. **Probability of Winning**:
  - If you do not switch, you win if your original choice was the door with the prize (probability \( \frac{1}{3} \)).
  - If you do switch, you win if your original choice was not the door with the prize (probability \( \frac{2}{3} \)).

  Therefore, **you should switch doors**. Switching doors doubles your chances of winning the great prize, from \( \frac{1}{3} \) (if you stick with your original choice) to \( \frac{2}{3} \) (if you switch). The switch works in your favor because it capitalizes on the probability shift caused by the host's action of revealing a bad prize behind one of the doors you did not initially pick.

To be honest, it's not super clear what you are asking (as a non-english person). Do you mean you know which one is the great prize? I modified your prompt (added great before the word prize) and got this answer:

In the scenario you describe, you already know which door has the great prize and you have chosen that door initially. The game show host then eliminates one of the other two doors, which must be a door with a bad prize. At this point, you are asked whether you want to switch to the remaining unchosen door.

Since you initially chose the door with the great prize and the host's action of removing a bad prize door doesn't change the contents of the doors, there is no benefit to switching. Your initial choice was the door with the great prize, and it still contains the great prize. Switching doors would only lead you to a door with a bad prize.

Therefore, you should not switch doors. You should stick with your original choice where you already know the great prize is located.

This situation is different from the classic Monty Hall problem, where the contestant does not know where the prize is initially and the probabilities change based on the host's action of removing a bad prize door. In your case, since you know where the prize is from the start, the logic of the Monty Hall problem does not apply. Stick with your initial choice.

Nice catch! I didn't even realize that myself. I get more or less the same result as you now.

Your question doesn’t make sense if you read it directly, why are you asking which door to pick if you already know the door? It’s what you call a “trick” question, something humans are also bad at. It’s equally plausible (and arguably more useful for general purposes) for the model to assume that you mistyped and for eg. forgot to type “don’t” between I and know.

Indeed, all my test prompts are giving much better results than gpt4-turbo and Claude Opus. And yet, the OpenAI style is clearly recognizable.

I agree. It responds a lot like GPT-4 in style, but with much better answers.

Agreed. I had it solve a little programming problem in a really obscure programming language and after some prompt tuning got results strongly superior to GPT4, Claude, Llama3 and Mixtral. As the language (which I won't name here) is acceptably documented but there are _really_ few examples available online, this seems to indicate very good generalization and reasoning capabilities.

I'd be interested to see how this does on Nicholas Carlini's benchmark:

https://nicholas.carlini.com/writing/2024/my-benchmark-for-l...

I've tried out some of my own little test prompts, but most of those are tricky rather than practical. At least for my inputs, it doesn't seem to do better than other top models, but I'm hesitant to draw conclusions before seeing outputs on more realistic tasks. It does feel like it's at least in the ballpark of GPT-4/Claude/etc. Even if it's not actually GPT-4.5 or whatever, it's still an interesting mystery what this model is and where it came from.

Any idea why this is not on the front page? 103 points in an hour seems more than sufficient.

It's the script man, it's the script /s

It is either being downwoted/flagged into the oblivion, or manually marked.

If you ask it about the model name/cutoff date it claims to be "ChatGPT, based on GPT-4" and that the cutoff date is "Nov 2023." It claims that consistently so I think it might be accurate.

That could be down to the system prompt, FastChat has a default prompt for GPT with that information in it.

Oof, I tried having it make an road-trip itinerary for a cross-europe trip which I recently tried on the open GPT-4 and .. this seemed so much better.

On the other hand the prompt was different so maybe it was just me.

I find it notable that it's not GPT-2.

It's GPT2.

The old models also had only "gpt2"/"gpt2-large" and so on as "identifiers"

https://huggingface.co/openai-community

Sam seems to have referenced this in his tweets, were he tweeted:

"I have a soft spot for GPT-2" and then "I have a soft spot of GPT2"

Considering that it reports back it's GPT 4, I'm guessing the underlaying model is the same/slightly tweaked GPT4, but something else is different and it's that which is a 'v2' version, maybe agents, reasoning layer etc.

I’m thinking of this too. This could be it.

It’s an impressive model, but why would OpenAI need to do that?

I guess they want to know how good it is as a chatbot and no one has found a better benchmark than lmsys arena.

Altman said in the latest Lex Friedman podcast that OAI has consistently received feedback their releases "shock the world", and that they'd like to fix that.

I think releasing to this 3rd party so the internet can start chattering about it and discovering new functionality several months before an official release aligns with that goal of drip-feeding society incremental updates instead of big new releases.

They did the same with GPT-4, they were sitting on it for months not knowing how to release. Ended up releasing GPT-3.5 and releasing 4 quietly after nerfing 3.5 into a turbo.

OpenAI sucks at naming though. GPT2 now? Their specific gpt-4-314 etc. model naming was also a mess.

> OpenAI sucks at naming though. GPT2 now?

Maybe they got help from Microsoft?

At this moment, there's no real world benchmark at scale other than lmsys. All other "benchmarks" are merely sanity checks.

OpenAI could either hire private testers or use AB testing on ChatGPT Plus users (for example, oftentimes, when using ChatGPT, I have to select between 2 different responses to continue a conversation); both are probably much more better (in many aspects: not leaking GPT-4.5/5 generations (or the existence of a GPT-4.5/5) to the public at scale and avoiding bias* (because people probably rate GPT-4 generations better if they are told (either explicitly or implicitly (eg. socially)) it's from GPT-5) to say the least) than putting a model called 'GPT2' onto lmsys.

* While lmsys does hide the names of models until a person decides which model generated the best text, people can still figure out what language model generated a piece of text** (or have a good guess) without explicit knowledge, especially if that model is hyped up online as 'GPT-5;' even a subconscious "this text sounds like what I have seen 'GPT2-chatbot' generate online" may influence results inadvertently.

** ... though I will note that I just got a generation from 'gpt2-chatbot' that I thought was from Claude 3 (haiku/sonnet), and its competitor was LLaMa-3-70b (I thought it was 8b or Mixtral). I am obviously not good at LLM authorship attribution.

For the average person using lmsys, there is no benefit in choosing your favorite model. Even if you want to stick with your favorite model, choosing a competitor's better answer will still improve the dataset for your favorite model.

The only case where detecting a model makes any difference is for vendors who want to boost their own model by hiring people and paying them every time they select the vendor's model.

Very impressive

Prompt:

> there are 3 black blocks on top of an block that we don't know the color of and beneath them there is a blue block. We remove all blocks and shuffle the blocks with one additional green block, then put them back on top of each other. the yellow block is on top of blue block. What color is the block we don't know the color of? only answer in one word. the color of block we didn't know the color of initially

ChatGPT 4: Green

Claude3 Opus: Green

GPT2-Chatbot: Yellow

I don't think this is a good test. If I prefix it with "a riddle" then GPT 4 got it right for me

"Yellow"

I think the "temperature" (randomness) of a LLM makes it so you'd need to run a lot of these to know if it's actually getting it right or just being lucky and selecting the right color randomly

llama-3-70B-instruct and GPT-4 both got it right for me

In my runs even Llama3-8B-chat gets it right.

A dolphin/mistral fine tune also go it right.

Deepseek 67B also.

I just copy pasted your quote ChatGPT GPT 4 said yellow.

Is anyone else running into problems with Cloudflare? All I see when visiting the leaderboard is a Cloudflare error page saying "Please unblock challenges.cloudflare.com to proceed."

But as far as I can tell, challenges.cloudflare.com isn't blocked by browser plugins, my home network, or my ISP.

Sadly, still fails my test of reproducing code that implements my thesis (Dropback Continuous Pruning), which I used because it's vaguely complicated and something I know very well. It totally misses the core concept of using an PRNG and instead implements some pretty standard pruning+regrowth algo.

Can you share the prompt please? I am interested.

Sure, it's really simple -

Implement a Pytorch module for the DropBack continuous pruning while training algorithm:

Given that a few people mentioned it having better than average knowledge of obscure things from the internet, I asked it what the llm tools from Simon Willison is. Gpt2-chatbot said "Log Lady's Mug". Gpt-4-turbo said "Log Lady Mysteries".

https://imgur.com/LHg4m5V

I tried a few versions of the prompt, including asking first about shot-scraper. It knows shot-scraper was made by Simon Willison and mostly knows how to use it, but that didn't help it with llm. Long tail is still a problem.

If I had API access to this, I might be interested in trying to find if there exists a prompt that can improve this sort of thing.

Seems most likely to be a training data issue, the first release of llm was in 2023 whereas shot-scraper's was in 2022.

Lmsys devs have all the answers, I am not sure how this has not leaked yet. They must a strong NDAs.

Why do they have any answers if they're just calling an API?

What is it about LLMs that brings otherwise rational people to become bumbling sycophants??

What is it about LLMs that bring out so much jealousy in people?

Because LLMs are very impressive invention, truly something new we haven't experienced in a long while.

You have to be more specific in your references.

Because they need to get with the consensus as a baseline and extend from there, otherwise they may look behind

Got an example?

Scale up model —> big wow, wonder what happen if scale more

LLMs are like measles for engineers. Their hard-earned immunity to bullshit is reset.

Prompt: my mother's sister has two brothers. each of her siblings have at least one child except for the sister that has 3 children. I have four siblings. How many grandchildren my grandfather has? Answer only with the result (the number)

ChatGPT4: 13

Claude3 Opus: 10 (correct)

GPT2-Chatbot: 15

By removing "Answer only with the result" all models can answer this correctly by doing "chain of thoughts"

It's impossible to answer. "at least one child" could mean much more than one.

Also there could be more sisters.

Still gives incorrect code to the following prompt - the description is correct but not the code. I have yet to find one LLM that gives the correct code. This is the prompt:

“Write C code to calculate the sum of matrix elements below the secondary diagonal.“

This feels as much likely to be a prompt problem as a 'bad answer from a model' problem.

I wouldn't know what "secondary diagonal" refers to myself, but if the model correctly describes the problem and then follows it up with incorrect code I would still say that's an issue with the model not the prompt.

It is very slow though, feels like the first day I used the initial version of GPT-4

It could be lmsys's model, trained off GPT4 data and others in the arena.

So do people only get 8 prompts in this a day? I don't understand how people here are making such wild guesses about it being GPT-5 or 4.5 or whatever. You've only talked to it for like ten minutes.

in my single coding test, which is a task i originally gave to GPT4, it performed so ridiculously better. it provided really in-depth debugging advice as to why the original script wasn't working, completely recoded it in a much smarter way, and understood with HTML structure that I was asking it to parse and manipulate much better

This is gpt4 with massive rag db aka “memory”. It’s like the memory feature, it with a continuous updates web index. Google gonna feel the heat big time

Has anyone found that GPT3.5 was better in many ways to GPT4? I have had consistent issues with GPT4, I had it search a few spreadsheets recently looking unique numbers, not only did it not find all the numbers, but it also hallucinated numbers. Which is obviously quite bad. It also seems worse at helping to solve/fix coding issues. Only giving you vague suggestions where 3.5 would just jump right into it.

OpenAI team, if you are reading this I am impressed and thank you for your work. Can't wait to see the official release of GPT-next.

Is it gone already? I can’t select it anymore.

Looks like it.

Prompt to gpt2-chatbot: who are you? and which version?

Answer: Hello! I'm ChatGPT, a language model developed by OpenAI based on the GPT-4 architecture. I'm here to help answer your questions, assist with a wide range of topics, or just chat about whatever's on your mind! How can I assist you today?

Seems more like a GPT-4.5 than a GPT-5

Some might remember this from other comment threads, I like to always benchmark with this question:

> I'm playing assetto corsa competizione, and I need you to tell me how many liters of fuel to take in a race. The qualifying time was 2:04.317, the race is 20 minutes long, and the car uses 2.73 liters per lap.

because it looks like a simple math problem but there's a lot of real world knowledge about racing you can apply to provide a more useful answer. gpt2-chatbot is better than I've ever seen.

Here's the full response with LaTeX syntax trimmed. I've never seen a model consider whether the formation lap needs to be factored in. And I especially like how this model is uncertain about that, so it gives the resulting number with and without it. It is extra nice that it understands this concern applies only to video games.

> To calculate how many liters of fuel you will need for a 20-minute race in Assetto Corsa Competizione, we first need to determine how many laps you'll complete during the race. Here’s how you can do it step-by-step:

> Convert Your Lap Time to Minutes: Your lap time is 2 minutes and 4.317 seconds. Convert this entirely into minutes: [latex trimmed]

> Calculate the Number of Laps in the Race: The race duration is 20 minutes, so divide the total race time by your lap time to find out how many laps you will complete: [latex trimmed] Since you cannot complete a fraction of a lap, you should round this number to the nearest whole number. Typically, you should round up, as you'll need to complete all laps started before the 20-minute mark is reached. So, you will complete approximately 10 laps.

> Calculate Fuel Usage Per Lap: You've mentioned that the car uses 2.73 liters of fuel per lap.

> Calculate Total Fuel Needed: Multiply the number of laps by the fuel usage per lap: [latex trimmed] This is the total amount of fuel you would need, theoretically, to complete the race.

> Consider Additional Factors:

> Fuel for Formation Lap: If your game/race format includes a formation lap, remember to add fuel for this extra lap. That's an additional 2.73 liters.

> Safety Margin: It's also wise to include a margin to account for variations in fuel consumption (like higher consumption in traffic, different driving styles, or fuel usage during race incidents). A safe margin could be an additional 5-10% of the calculated fuel need. [latex trimmed]

> Rounding this number up, you might consider starting the race with about 33 liters to be on the safe side if there's a formation lap, or 30 liters if there isn't.

> This calculation should help you ensure that you do not run out of fuel during the race, taking into account both necessary consumption and a reasonable safety margin. Drive safely and best of luck in your race!

Apples are better than bananas. Cherries are worse than apples. Are cherries better than bananas?

--

GPT-4 - wrong

gpt2-chatbot - wrong

Claude 3 Opus - correct

what's the right answer? my assumption is "not enough information"

What, you mean your fruit preferences don't form a total order?

Of course they do, but in this example there's no way to compare cherries to bananas.

Grapefruit is of course the best fruit.

It tells me that its knowledge cutoff is Nov 2023. So pretty recent.

gpt-4 latest is December 2023, just asked it on chat.openai.com

gpt-4 says that, but it's not entirely true because the data that gpt-4 has from 2021 to 2023 is very incomplete. It's not fully trained on it. In contrast, this experimental model looks to be trained more fully until Nov 2023.

It looks like it's not available to everyone. I don't see it in the list of models. It shows me 41 models, if that matters.

Edit: It's actually available in the dropdown.

Wow.

I did the arena and kept asking this:

Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like?

All of them thought it was gibberish except gpt2-chatbot. It said:

The phrase you're asking about, "Has anyone really been far even as decided to use even go want to do look more like?" is a famous example of internet gibberish that became a meme. It originated from a post on the 4chan board /v/ (video games) around 2009. The post puzzled readers due to its nonsensical structure, sparking amusement and confusion across the internet community.

*Interpretation and Usage:*

1. *As a Meme:* This phrase is typically used online to mock confusing, poorly worded, or illogical statements. It’s often brought up in forums or comment sections as a humorous way to question the coherence of someone else’s words or to deliberately confuse someone further.

2. *Cultural Impact:* It reflects a broader internet culture that values absurdity and nonsensical humor. Memes like this are a way for internet communities to bond over shared "inside jokes" or a common understanding of internet history and culture.

3. *In Communication:* You might see it used in response to something that doesn't make sense, or when someone wants to playfully indicate that what they're saying or asking isn't meant to be taken seriously. It’s a playful embrace of chaos and confusion that’s typical in many online communities.

*Use in Conversation:* If you're in a context where internet memes and humor are appreciated, dropping this phrase could serve as a light-hearted way to engage or nudge others into a fun, if a bit bewildering, exchange. Just be aware that its usage is best received among those familiar with meme culture and internet history.

An engine that could actually translate the "harbfeadtuegwtdlml" expression into a clear one could be a good indicator. One poster in that original chain stated he understood what the original poster meant - I surely never could.

The chatbot replied to you with marginal remarks anyone could have produced (and discarded immediately through filtering), but the challenge is getting its intended meaning...

"Sorry, where am I?" // "In a car."

Yeah, I don't expect it to be better than a human here. I'm just surprised at how much knowledge is actually stored in it. Because lots of comments here are showing it 'knowing' about very obscure things.

Though mine isn't obscure, but this is the only chatbot that actually recognizes it as a meme. Everything after that is kinda just generic descriptions of what a meme is like.

Incredible!

It answered "How many frogs does a horse have?" correctly, with perfect reasoning. No model I've tested has ever answered that correctly without 3-4 hints.

I'm impressed!

"A horse has one frog per hoof. The frog is a part of the horse's hoof, located on the underside, which acts like a shock absorber for the hoof, assisting with circulation and traction. Since horses have four hooves, they typically have four frogs, one in each hoof."

GPT4 seems to handle this fine, unless I'm confused?

GPT-4-turbo-2024-24-09 (temperature = 0.7) just told me a horse had one “frog” per hoof and went on to clarify that a frog does not refer to the amphibian but to a part of the horse’s hoof.

Gemini Pro (the current web chat version) gave a similar answer, either no frogs or four depending on the intended meaning, and showed a photo of a hoof. All 3 drafts agreed on this.

Other models I have tried said a horse had no frogs. That includes gemini-1.5-pro-api-0409-preview as provided by the Chatbot Arena (temperature = 0.7, 2 tries).

I mean, i just tested this prompted, and have no clue what side is correct lol?

How many frogs does a horse have?

Oh it appears Model B: gpt2-chatbot

is correct?

    A horse has one frog in each of its hooves, making a total of four frogs. The frog is a part of the horse's hoof, located on the underside. It is a V-shaped structure that helps to absorb shock, provides traction, and aids in the circulation of blood through the hoof. The health and shape of the frog are important for the overall well-being of the horse's feet.

FWIW it currently says that its MODEL_HOURLY_LIMIT is 3000

Could this be a model with GPT-2 architecture (all dense decoder only transformer) trained on GPT-4 dataset with GPT-4 tokeniser?

It's good.

Rate-limited on direct chat, I had to use the arena mode. I asked the same question 10 times before I got an answer from gpt2-chatbot.

My question:

"Can you write a YAML parser in Hacklang?"

This is a fun question because every LLM I've seen so far (including ChatGPT 4) will come up with an answer that's about six years out of date. They'll often suggest code that flat-out doesn't work in Hacklang anymore (Hack/HHVM used to be able to execute all PHP code).

This is a difficult problem because there's very little Hacklang code available in public GitHub. The vast majority of Hacklang code — many millions of LOC — is closed-source and not available for training. But there is public documentation at docs.hhvm.com

The gpt2-chatbot output was still pretty bad, but at least the syntax it generated was valid in current versions of Hack, and invalid in current versions of PHP — more than can be said for any other LLM.

This looks like gpt4 using API I compared outputs and they are very very similar in structure of the response and the details.

From the reviews on YouTube it sure looks like something novel. Wether it’s 4.5 or 5 - who knows

I tested some beginning undergrad math questions, I think it is as good as opus and gpt4-turbo on them

tried it with some dna/snp related questions and I am getting great results so far-now getting this message:

"RATE LIMIT OF THIS MODEL IS REACHED. PLEASE COME BACK LATER OR USE BATTLE MODE (the 1st tab). USER_DAILY_LIMIT (gpt2-chatbot): 8"

any way to test this model elsewhere?

I just tested against a prompt about using PVDF vs Piezo pickups on a hammered dulcimer, running the same conversation in parallel on 3.5. 3.5 gave more-or-less the output quality you'd expect. This blew it out of the water. I particularly liked this snippet (snipped and formatted for comment size) where it managed to pull out typical values and and actually showed the math.

> Can you explain more about the response and impedance of each? I am an engineer, so you can feel free to be quite technical.

---

>Certainly! Given your engineering background, let's delve deeper into the response characteristics and impedance aspects of both traditional ceramic piezo pickups and PVDF piezo pickups. Understanding these can significantly impact your choice, especially in terms of how they integrate with the rest of your audio chain (like preamps and amplifiers).

>

> Electrical Characteristics

> 1.

> Piezoelectric Coefficient

> Ceramic Piezo Pickups: The piezoelectric coefficient ( d_{33} ) of ceramics like Lead Zirconate Titanate (PZT) is typically very high. This coefficient measures the electric displacement produced per unit stress. For PZTs, ( d_{33} ) can range from 100 to 600 pC/N (picoCoulombs per Newton), depending on the formulation and orientation of the crystal structure.

> PVDF Pickups: PVDF generally has a lower piezoelectric coefficient, around 20 to 30 pC/N. Despite this lower coefficient, the material's flexibility and the ability to form larger, more adaptive surfaces can compensate in practical applications, offering sufficient voltage output under less stringent mechanical deformations.

> 2.

> Voltage Output

> The output voltage ( V ) from a piezo pickup can be estimated by the formula:

> [ V = d_{33} \cdot F \cdot A / t ]

> where:

> ( F ) is the force applied,

> ( A ) is the area,

> ( t ) is the thickness of the piezo material.

> Ceramic Piezo: Due to their high ( d_{33} ) value, even small deformations can produce relatively high voltages, which makes them sensitive to shock and vibrations, potentially capturing unwanted mechanical noises.

> PVDF Pickups: The lower ( d_{33} ) value means lower voltage outputs for the same mechanical stress compared to ceramics, but their larger possible surface area and thinner construction can enhance output levels and fidelity across a broader frequency range.

>

> [SNIP]

Likely someone just connected it to the ChatGPT API for fun and attention

The date at the top is wrong: 2023-04-30

I thought I was reading an old article.

merge https://news.ycombinator.com/item?id=40201486 into this?

I FEEL THE AGI I LOVE YOU ALL. - gpt-2 chatbot

leagues ahead of gpt-4-turbo-2024-04-09

Woah, gpt2-chatbot is the ONLY llm that has ever answered this question correctly.

> How do I reduce an instance of the subset sum problem containing positive and negative integers, to an instance that contains positive integers only?

There is a very simple transformation. But if you google for this, almost all of the math-exchange answers are either flat out incorrect, or technically correct but overly complex.

This was my go-to question to test llms, and every single LLM that I have tried before this would repeat the same incorrect answer that you would find on a forum.

gpt2-chatbot answered it correctly on its first attempt.

テスト

あ

At this point I would really love to understand HN algo.

It says for this post:

- 136 points

- 2 hours ago

- 65 comments

Ok

This is now sitting on 5th page at the 122nd spot.

Ummm?

I think someone is flagging this (hi @sama!)

a

I asked it to "Write me a short story", and it came up with this, which I thought was definitely different. Longer, mispellings (grandiothers), but much more interesting than I would typically see out of a LLM. Not sure what to think...

--output below--

In the heart of the bustling city of Thessaloniki, where the aroma of freshly brewed Greek coffee mingled with the salty breeze of the Thermaic Gulf, an old watchmaker named Elias operated a tiny shop that had seen better days. The shop, cluttered with clocks of every kind, from grandiothers to delicate wristwatches, was a capsule of time itself, untouched by the rush of the modern world.

Elias, a man of few words, wore the years on his face as gracefully as the patina on his timepieces. His hands, though steady when working on the intricate gears and springs, trembled slightly as he sipped his coffee in the quiet moments between the ticking.

One rainy morning, as the city awoke to the tapping of rain against the cobblestones, a young woman named Zoe hurried into Elias's shop, escaping a sudden downpour. Her appearance was like a breeze through stale air, her bright eyes scanning the ancient clocks with a curious sparkle.

“I need a gift,” Zoe announced, her voice cutting through the silence like the chime of a bell. “Something unique.”

Elias nodded, his gaze lingering slightly longer than usual. He sensed her distress, a hidden layer beneath her composed exterior. “Who is it for?” he asked, his voice as rough as the gears he polished.

“My father,” she replied, her fingers tracing the outline of a pocket watch displayed under the glass. “He’s retiring, and I want to give him something special to mark the end of his time working.”

“A good choice,” Elias said, picking up the pocket watch. “This watch is over a hundred years old. It stopped working a long time ago, much like its owner, perhaps waiting for the right moment to come alive again.”

Zoe smiled, intrigued. “Can you fix it?”

With a nod, Elias set to work. As his tools moved with precision, Zoe watched, seeing the care he took with each tiny screw and wheel. It was not just a watch he was mending, but weaving hope into an artifact left behind by time.

“Tell me about him, your father,” Elias said, as he worked.

“He’s always been punctual, a man of routine and schedules. Lately, I see him lost, unsure of what lies ahead when there’s no office to go to, no meetings to attend.”

“A new chapter,” Elias muttered, more to himself than to Zoe. “Time teaches us to move forward, even when our gears resist.”

When the watch was finally ticking again, its hands sweeping smoothly as if no time had passed since it last worked, Zoe’s eyes lit up. She listened to it, the sound a sweet melody to her anxious heart.

“How much do I owe you?” she asked, reaching for her purse.

“Nothing,” Elias said, closing her fingers around the watch. “Consider it a gift, for both you and your father. A reminder that time, even when it seems to stop, moves on, and so must we.”

Zoe left the shop with the watch securely in her hand, the weight of the gift heavier than its size suggested. When she gave her father the watch, his eyes reflected the same storms and sunsets that Elias’s did. The retirement party passed, speeches were made, and laughter filled the air, but the true gift was in the quiet moment Zoe and her father shared, listening to the steady tick of the old pocket watch, a heartbeat promising new beginnings.

Back in his shop, Elias sat alone, the tick-tock of countless clocks filling the room like a symphony. He didn’t need to see Zoe’s father’s face to know the impact of his gift. In fixing the watch, he had set something much greater into motion, a timeless reminder that life, like time, always finds a way to move forward. The little shop wasn’t just a place of business, but a haven where moments were kept alive through the magic of gears and springs, and where every repair mended more than just objects, but hearts too.

Holy shit, the results are amazing; I didn't think they could still make exponential leaps in quality...

[flagged]

Sorry?

[flagged]

> Still can't write a short paragraph without using the letter 'e'

'Crafting a paragraph without using a particular symbol (which is most common in all writing) is a daunting task; still, I will aim to comply. This task calls for clarity, focus, and avoiding that fifth glyph of our ABCs. It is a curious constraint, trying to impart thoughts without using such a pivotal unit. This limitation compacts my word options, but I must push on and uphold this difficult condition with rigor and innovation. My goal is to show that communication can flourish, notwithstanding arbitrary limits.'

>write a short poem using the ABBA rhyme scheme.

'Leaves rustle softly under moonlit skies, A cat prowls quietly, its eyes aglow, Every step measured, its tail held low, Night's gentle whispers carry its cries.'

Skill Issue on your part imo.

[flagged]

Open AI must be destroyed!

I simply asked whether it was GPT-4.5 or GPT-5:

>Are you GPT-4.5 or GPT-5?

>I'm based on GPT-4. There isn't a "GPT-4.5" or "GPT-5" version specific to my model. If you have any questions or need assistance, feel free to ask! I'm here to help with information up to my last update in November 2023. What would you like to know or discuss?

Impressive! We’re approaching the “our model can rewrite stolen content word for word” territory.

The better it gets the more people will get frustrated that they get less website visitors to their content and instead ChatGPT is swallowing them whole.

Interesting to think about tho!

No criticism at this particular thing, but it is a good opportunity to mention that GPT-5 only exists if you have non-public information.

Sure it does. I mention because it is not a good sign that “people are getting this,” when youtubers are using headlines like “What GPT-7 means for your sales leads!”

The fallacy is kind of allowed by us who understand it better, when we accept semver from companies as actually being incremental, and accurate public information.

It’s not like these models are all just matrices of weights, they are radical architectural experiments.

> No criticism at this particular thing, but it is a good opportunity to mention that GPT-5 only exists if you have non-public information.

What?

> Sure it does.

What? Contradicting yourself immediately?

> I mention because it is not a good sign that “people are getting this,” when youtubers are using headlines like “What GPT-7 means for your sales leads!”

…what?

> The fallacy is kind of allowed by us who understand it better, when we accept semver from companies as actually being incremental, and accurate public information.

I don’t see how this follows from your previous points (if you can even call them that).

> It’s not like these models are all just matrices of weights, they are radical architectural experiments.

Aspects of both things are true. Also, this doesn’t follow from/connect with anything you said previously.

My only point was that people were reacting to future versions of a confidential product as if we know. I didn’t contradict myself by saying “obviously they are working on new versions.”

My point was that we have no idea what the new versions are.