Dr Kamran SoomroSep 12, 2024

Recently I’ve been testing *Claude 3.5 Sonnnet’s* capabilities to assist with writing code for ML experiments.

The task was simple. Claude was asked to do the following:

1) Load an image dataset from a local path using the *datasets* library.

2) Create an embedding signature of the dataset using the fastai library.

Show threadDr Kamran SoomroSep 12, 2024

The code it generated was reasonably simple. Some of the API calls were deprecated but they were easily fixed. The expected structure of the object returned by the *load_dataset* method was also incorrect. I manually fixed that too. However, the major problem was that there was another error while loading the dataset.

I shared the error with Claude and it reworked the code by adding some logging output to debug.

A reasonable strategy

Show threadDr Kamran SoomroSep 12, 2024

Essentially the problem was that when loading the dataset, some of the images were returned as paths, while others were returned as image objects. Claude’s code was only expecting the former.

I wanted to test Claude’s reasoning capabilities so I shared the debug output with Claude. It tried reworking the code but the changes it was making made no sense and did not address the actual problem.

Show threadDr Kamran Soomro

In the end it was a simple fix to get the code working. But at least to me it’s quite clear that what Claude is doing is not actual reasoning but simply mimicking common solutions to common problems.

This is not to say it’s not useful. It did save me the trouble of looking up most of the API calls myself. But it is definitely not ready to be autonomous. Oversight by a reasonably skilled programmer is still required and recommended.

Sep 12, 2024 at 3:44pmWeb
Show threadDr Kamran SoomroSep 12, 2024
The thing I worry about though is the effect it will have on the development of critical thinking skills in new learners. It’s too easy to use these tools as crutches and try and delegate your thinking to them. And if new learners don’t spend the time to struggle with the nitty gritty of the basic concepts, they won’t develop the high level expertise required to effectively orchestrate and deploy these tools in practice.
Show threadBen JudsonSep 12, 2024
@ksoomro best case scenario, a smart learner can use these tools to focus their energy on core concepts, patterns, and strategies, and spend less time on the quirks and APIs of individual languages. But in general it seems that the effect tends to be making experienced coders a bit faster, rather than helping new folks learn to code.
Show threadDr Kamran SoomroSep 12, 2024
@shiftingedges thats exactly it. But also it’s preventing new coders from acquiring the necessary skills to do the former.
Show threadAdrienBufortSep 14, 2024
@ksoomro Personnally I wouldn't be that worried. It's like Platon saying that writing will make us lazy thinker because we could delegate memory function to script for example. My guess is that people will adopt LLMs like people adopt script/writing in the past : it will probably change the way we think (compare oral culture vs "writing" culture).