College technical AI safety hackathon retrospective - Georgia Tech

TLDR: the AI safety initiative at Georgia Tech recently hosted an AI safety focused track at the college's flagship AI hackathon. In this post I share how it went and some of our thoughts.

Overview

Hey!  I’m Yixiong, co-director of Georgia Tech’s AI safety student group.  We recently hosted an AI safety focused track at Georgia Tech’s biggest AI hackathon, AI ATL.  I’m writing this retrospective because I think this could be a useful data point to update on for fellow AIS groups thinking about hosting similar things!

The track was focused on evaluations on safety-critical and interesting capabilities, this is the track page that was shown to hackers (feel free to reuse/borrow content, just let us know!)

Huge thank you (in no particular order) to Michael Chen, Long Phan, Andrey Anurin, Abdur Raheem, Esben Kran, Zac Hatfield-Dodds, Aaron Begg, Alex Albert, Oliver Zhang, and others who helped us make this happen!

Quick stats:

• ~350 hackers (overall hackathon).

• 104 projects submitted (overall hackathon).

• Submissions to our AI safety track: 16 teams (~50 people).

  • 6 projects were solid/relevant, the rest were very noisy submissions, since you could submit to as many tracks as you want to.

• Estimate # of low/moderate engagement with AI safety (attending workshops, reading track information): 100 hackers.

• Estimate # of medium/high engagement with AI safety: 20 hackers.

• These are the submissions that we got, the 6 solid projects were (in order of goodness, keep in mind that most of these came from people who were new to AI safety!):

  • Privacy-Resilience and Adaptability Benchmark (PRAB): puts the model in a realistic and sensitive deployment environment and benchmarks models against several categories of prompting attacks

  • StressTestAI: similar to the above, but in less realistic and ‘higher stakes’ settings like disaster response, but creative metrics.

  • DiALignment: benchmarked refusal after performing activation steering away from the refusal behavior.

  • AgentArena: set up agents in cooperative games (like prisoner’s dilemma) and observed behavior

  • Are you sure about that?: tried to benchmark LLMs’ ability to spot unfaithful CoT against humans (the user)

  • LLM Defense Toolkit: set up a pipeline to benchmark the safety of a user specified LLM with an array of generated attacks.

Relevant track details:

We tried to optimize the track in a bunch of ways, including but not limited to:

• Competitive prize (cash is per team): 

  • 1st place: $400 cash + auto acceptance to AISI’s AI safety fellowship next spring

  • 2nd place: $200 cash + auto acceptance to AISI’s AI safety fellowship next spring

• Association with big names: we listed Anthropic and Apart Research as supporters

  • Anthropic gave the track a shout out next to their other track “build with Claude”

• Approachableness: made track description as non-intimidating as possible, providing an abundance of support in the form of mentorship and workshop/speakers

• Appeal to intellectual curiosity: 

  • Background reading for AI safety/evaluation fundamentals ~30 min total

  • Events: about 30 people attended each one.

    • Workshop by Apart research: how to scaffold LLMs, build agents, and run evaluations against them

    • Talk by METR: the case for AI evaluations and governance

    • Talk by CAIS: jailbreak and red-teaming LLMs

Execution

As with all events, execution matters a lot.  This is the area that we felt could be improved the most.

• Collaboration: do a vibe check if you’re thinking about collaborating with your main hackathon org on campus! 

  • We chose to host this as part of a general AI hackathon (rather than standalone) hoping to leverage the main host’s organizing capacity and reach to expose new people to AI safety.  This wasn’t a great experience for us mainly because the main hosts never really tried to understand what our track was about (probably faults on both sides). The impression is that it was a chore to deal with us, so make sure they’re on board before collaborating! You shouldn’t over-update on this, we may just have an outlier.

• The contents of the track were well calibrated for difficulty and perceptiveness, as per feedback from teams that gave our track a shot and properly engaged with the topics.

  • You should try to have them before hacking begins. A complaint is that speakers/workshops take time away from hacking.

  • Great feedback from hackers who read through our materials and gave it a shot

• Physical presence: I think we could’ve gotten double the number of solid submissions if we had a significant physical presence at the in-person venue.

  • What we did

    • Have mentors available in person and online during office hours

    • Project virtual speaker events onto a screen in a physical room and announce them in person.

  • What we wish we also did

    • Have a booth/table on day 1 of the hackathon when everyone comes to check in.  Give away stickers/merch and pitch our track

    • In person speakers, especially from big name companies.

    • Wear AI safety club merch (although we don’t even have merch…)

Our opinion/takes

• Hackathon patterns: hackathons are a staple at major technical universities.  These may be well known, but I had never attended a hackathon before this and found these interesting.

  • The BEST time to pitch your track and make announcements is the first day when people come for check in, since everyone is there.

  • Do the convincing (speaker, workshop, etc) before the end of the first day, since people usually decide which track to do/their idea by then.

  • Best time for in-person events (when people will be at the venue): first day during check in and right after food is served…

• Potential failures modes / most valuable things to improve

  • If you can, try to communicate that popular tools/libraries/frameworks are useful for your track!

    • People want to use their existing stack, and thinking they have no comparative advantage in anything new to them.

    • Probably the main shortcoming, despite our track being by far the most interesting (the rest were like “best use of XXX”...)

  • The track being incompatible with other tracks if submission to multiple tracks is allowed.  Going for the safety track means losing out on the others.

  • People being confused on what to do

    • Make sure you explain clearly what you want people to do as this is a niche topic for now, give example projects (see ours) and starter code.

    • People do NOT like reading!  Maximize the signal to words ratio!!!

  • You should borderline spam announcements, contact hackers to pitch EARLY, like before the hackathon starts if you can get their emails.

    • People have to know that your track exists!

  • Get a notable company to officially sponsor and a notable judge, this would be very difficult but will probably be the biggest attractive factor…

  • Do NOT have more than 3 tracks if you host a standalone hackathon, choosing is hard for people :)

• There is value in hosting at a general hackathon

  • The distribution of people at a general hackathon is different from the distribution of people who will come if you advertise a standalone AI safety hackathon.  If your goal is to reach new audiences, then being a part of a general hackathon will increase your chances of nerd-sniping!

• You should frame your AI safety specific workshops as useful for all the tracks, and appeal to credibility as much as you can.  Also very important to announce them in real time in person as people do NOT check slack/discord announcements.

What’s next?

I think our attempt serves as a successful proof of concept for bringing the topics of AI safety/alignment hackathons to campuses.  People will engage with the topic if you try really hard.  Don’t hesitate to reach out for help if you’re thinking of something similar and want to learn more about what we did!

Two things we might do in the future:

• Iterate on this and host a track at Georgia Tech’s data science hackathon

• Become an Apart Research node for hackathons and host standalone AI safety hackathons.

Thanks for reading and I hope it wasn’t too noisy!

Yixiong & the Georgia Tech AISI team.