Christopher Caulfield

🎮 **Harnessing LLMs in Video Game Development: A Quick Guide** 🕹️ Leveraging Large Language Models (LLMs) can transform video game narratives and interactions. Here’s a concise roadmap for product managers: 🔎 **Phase 1: Discover & Define** - **Market Insights**: Gather player insights on narrative preferences. - **Problem Identification**: Identify issues in storytelling or NPC depth. - **PRD Creation**: Outline LLM usage for dynamic narratives in a PRD. 🛠️ **Phase 2: Plan & Design** - **Select the Ideal LLM**: Choose LLMs based on dialogue capabilities. - **Prototype**: Test NPC dialogues via prototypes to refine AI roles. - **Feedback Loops**: Adjust AI strategy based on player feedback. 🔨 **Phase 3: Build** - **Prompt Engineering**: Craft prompts for coherent game-aligned content. - **RAG**: Use game lore to inform narrative content generation. - **Fine-tuning**: Enhance model responses with targeted data. ▶️ **Phase 4: Implement & Evaluate** - **Integration**: Seamlessly integrate LLM outputs into the game engine. - **Evaluate**: Use metrics like player interaction rates to assess AI impact. - **Iterate**: Refine AI elements based on ongoing feedback. 📊 **Phase 5: ROI Assessment** - **Cost Analysis**: Monitor expenses related to LLM operations. - **Performance Metrics**: Track player retention and in-game purchases. 🚀 **Phase 6: Launch & Optimize** - **Launch**: Promote AI-driven narrative features. - **Optimize**: Update the model based on player trends. 🎯 **Conclusion** Product managers can use LLMs to enhance gameplay and narrative depth, achieving significant engagement boosts. Dive into AI to redefine gaming experiences! 🌟👾 #AI #LLM #GamingIndustry #ProductManagement #VideoGames #Innovation #TechInGaming

24

2 Comments

Senior engineers lose money and responsibility at Meta partly because of an unwritten rule. Candidates are expected to not follow their interviewer's instructions to SUCCEED. Yes, you read that right. That blew my mind. There are other common reasons for downleveling as well. Some of these are expected but others are surprising. Kevin Landucci + Kashish Singhal, loved the simple practical takeaway for candidates. And the full episode rocked! Subscribe to ApplyPass to get more tips in your feed 🛎

20

3 Comments

Reader feedback for my latest post about how to use AI and my PRD template to never have to write a PRD from scratch again. Read the full post here: https://lnkd.in/eJjVXJUJ

8

The perfect idea doesn't exist. No idea turns out exactly how you expect it to when it confronts reality. As Naval shares, quality products come out of confronting reality 100x by putting something in the hands of customers and continually tinkering with it. 📌 I shared 9 more tips to elevate product quality here: https://lnkd.in/gRzMGmNE

13

3 Comments

Stop using the STAR for eng leadership interviews. When I landed my first public company VP Product offer, I had to learn the hard way that STAR isn't enough. Leadership interviews require a totally different approach. ICs are evaluated on execution skills. Leaders are also assessed on strategic thinking, business impact, and influence. 🧠💰🤝 The questions are different too: IC: "What's your superpower as a PM?" Leader: "How have you built a data-driven culture?" To crack leadership interviews, your stories need to: ✅ Explain the "why" behind your actions ✅ Highlight the "so what" of your results ✅ Show how you grew as a leader I developed the Triple-STAR framework to address these gaps. Triple-STAR takes STAR to the next level by adding: 💡 Stakes: Why the situation mattered to the business 💼 Status: Your specific leadership role and scope 🤔 Reflection: What you learned about leadership 🔁 Repetition: How you applied those lessons to drive further impact 🔗 Relating: How your experience maps to the new role I've used Triple-STAR to land multiple executive roles and coach countless leaders. It works because it showcases your ability to think strategically, drive business outcomes, and scale your influence. Don't undersell yourself in leadership interviews. Upgrade to Triple-STAR and tell stories that highlight your true executive potential. 🌟🌟🌟 Thoughts on how IC vs leadership interviews differ? Let me know in the comments. I wrote a deep dive guide with lots of examples for product, design, and engineering leaders from Director to CPO in the article.

16

3 Comments

I wrote Hiring Product Managers to show that hiring can be an agent of change. We need to stop thinking of hiring as simply 'ticking a box' - approaching hiring in a meaningful way can have a great impact on our personal growth, team dynamics & the shaping of our organisational culture. Get your copy!

10

I recently picked up "A D*mn Fine Stable Diffusion Book" by Manning Publications Co. (in early access) and I'm quite enjoying it so far. What I like most about it is that it isn't fluff: you're learning not just the basics, and you're not being force fed hype. You're learning how best to use Stable Diffusion and how it works. There are only chapters (of a planned 17) right now, so I'm eager for the new chapters to come down the pipe. https://lnkd.in/evt-WBKd

5

1 Comment

How can product leaders move the needle and take the lead in this shift? Moving Off the Solution – guide your team to step back from the tech stack and view the challenge from the POV of a new user. Measuring Customer Perceived Value -- require PMs to sit with a cohort of users -- typical users. Conduct a discovery exercise. A small sample size is sufficient. Embrace the Change – this should require a complete change in mindset but a different orientation. Focusing the team on adoption, cross sell and upsell has clear near-term and long-term ROI for the company. Learn more: https://lnkd.in/gjJk_K5V #leadership #prodmgmt #ROI

6

1 Comment

Free resource for anyone interested in leveling up on #AI and LLMs, especially Product Managers. Try this open chatbot on Hugging Face https://lnkd.in/gUsggM-T Example: To an audience of PMs, explain "Temperature" of LLMs, and its impact on the user experience

2

Here's a verbatim review from last month's cohort: "Tom's class is the only one on Maven thus far that I wholeheartedly rate a full 10 out of 10. Through fun assignments, we got hands-on practice on dealing with ambiguous, high-stake reviews. Beyond product reviews, Tom's extensive insights have been enormously helpful for a mid-career PM like me." I've got a new cohort starting in a few days for my 5-star rated, interactive, live, small-group workshops. Learn more and enroll here https://lnkd.in/gFFuUYpu #productmanagement

10

Interesting thing I've learned about building a benchmarking tool is that what a lot of early adopters actually want is an influencing tool. Completely reframes product features. Example: Old feature: Auto generated board reports 1. Choose metrics 2. Generate Graphs 3. Download as pdf/ppt New feature: Show me the stories I could tell with my data. 1. Here are the stories you can tell 2. Choose story 3. Generate metric benchmark graphs that help tell that story 4. Suggest narrative 4. Download Excited to see where peoplemetrics.fyi goes :-)

30

12 Comments

I recently conducted a mock interview for Product Design/Sense case with a talented candidate preparing for a Sr PM role for Meta. During our session, a few key areas of improvement emerged that I believe can benefit anyone preparing for their next big interview: 1. Target Unique User Segments: Rather than focusing on broad demographics or geographical segments, dig deeper into the motivations of your users. Understanding the "why" behind user actions allows you to craft more tailored and impactful solutions. For instance, segmenting users based on their motivations can lead to more meaningful insights and strategies. 2. Leverage the Customer Journey: Use the customer journey map to identify a comprehensive set of pain points for your prioritized user segment. This holistic approach ensures you're addressing all potential hurdles your users might face, leading to more robust and effective solutions. 3. Align with Business and Company Goals: Throughout your case, keep both the overarching business goals and the unique objectives of the company in mind. This alignment not only showcases your strategic thinking but also demonstrates your understanding of the company’s specific needs and how you can contribute to their success. By focusing on these areas, you can enhance your interview performance and showcase your ability to think strategically and empathetically about user needs. If you are correctly preparing for your product management interviews/thinking of starting your prep, feel free to DM us to get personalized help! Best of luck to everyone on their interview journeys! #InterviewTips #UserSegmentation #pmjobs #CustomerJourney #BusinessGoals #CareerAdvice #productmanagement #frameworks

2

Struggling to drive org-wide adoption for your research repository? If I could go back in time, this is what I would tell myself about scaling repositories and driving adoption… When it comes to onboarding teammates and stakeholders for any type of research repository, it's helpful to think of your users as 'Contributors' or 'Viewers'. Contributors are people who are conducting research. Viewers are the individuals who consume insights to help with decision making – but won't be adding knowledge to your repository. For Contributors, focus your onboarding around getting these users to the 'aha moment' realised by adding knowledge into your repository and discovering how their work fits into the bigger picture. In my experience, the faster a contributor experiences first-hand the added clarity (alongside efficiency) that a repository can bring, the more likely they are to adopt it. Don't force users to add all their research from day one. Instead, focus on helping them experience the system's benefits first, making them more likely to add more over time. For Viewers, focus your onboarding around enabling these users to discover insights that are relevant to the problems and challenges they're looking to solve today. Instead of asking Viewer users to perform irrelevant searches to ‘test the search function,’ inquire about their current focus. Use this information to search and query the repository for relevant insights. Again, focus on helping users experience the benefits of the system. Focus is key when it comes to onboarding and adoption for any research repository, and I’ve seen so many repository launches fall flat without it. By concentrating on the core value each user type experiences, you can design an effective onboarding process that maximises success and encourages organic growth through user invitations to your repository. Regardless of user type, I’ve found it’s best to start somewhat unscalable. For example with 1-1 / group calls (you’ll learn ton of insights about the onboarding process, as well as the repository itself). Next, consider building a simple onboarding guide or help centre, Loom videos and Slack channels are also super helpful. Time-box your effort, get something out there, reflect on the feedback, then iterate and launch the next version. Done is better than perfect. Keep pushing, keep iterating, and the adoption will follow 💪

3

1 Comment

AI at Meta has released the most powerful open-source model yet as of today: 𝑳𝑳𝑨𝑴𝑨3 - 8B and 70B models at 8k context length! They highlight improvements in each of the following aspects as the key differentiator: a) Model Architecture b) Pretraining data c) scaling up pre-training d) Instruction fine-tuning 💥 𝐏𝐫𝐞𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 & 𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐮𝐩: ✈️ Vocab of 128k tokens & GQA across both 8B and 70B, 8k native context length ✈️ Pretrained over 15T tokens, 4 times more code than LLAMA2 ✈️ Multi-lingual : over 5% of pretraining data consists of high-quality non English data that covers over 30 languages: don't expect same performance as English ✈️ LLAMA2 generates training data (like in Self-Instruct) to generate text classifiers that filter data powering LLAMA3 ✈️ Both 8B and 70B improve even after the model is trained on 2 orders of magnitude more data i.e log-linear improvement after 15T tokens training ✈️ Training runs on 2 custom-built 24k GPU clusters with an overall training time efficiency of 95% ✈️ 𝘖𝘷𝘦𝘳𝘢𝘭𝘭, 3 𝘵𝘪𝘮𝘦𝘴 𝘮𝘰𝘳𝘦 𝘦𝘧𝘧𝘪𝘤𝘪𝘦𝘯𝘵 𝘵𝘩𝘢𝘯 𝘓𝘓𝘈𝘔𝘈2 💥 𝑰𝒏𝒔𝒕𝒓𝒖𝒄𝒕𝒊𝒐𝒏 𝑭𝒊𝒏𝒆𝒕𝒖𝒏𝒊𝒏𝒈: ✈️ Combination of SFT + PPO + DPO + Rejection Sampling ✈️ 𝑸𝒖𝒂𝒍𝒊𝒕𝒚 𝒐𝒇 𝒑𝒓𝒐𝒎𝒑𝒕𝒔 𝒖𝒔𝒆𝒅 𝒊𝒏 𝑺𝑭𝑻, 𝒑𝒓𝒆𝒇𝒆𝒓𝒆𝒏𝒄𝒆 𝒓𝒂𝒏𝒌𝒊𝒏𝒈𝒔 𝒖𝒔𝒆𝒅 𝒊𝒏 𝑫𝑷𝑶, 𝑷𝑷𝑶 𝒉𝒂𝒔 𝒂𝒏 𝒐𝒖𝒕𝒔𝒊𝒛𝒆𝒅 𝒊𝒎𝒑𝒂𝒄𝒕 𝒐𝒏 𝒎𝒐𝒅𝒆𝒍 𝒑𝒆𝒓𝒇𝒐𝒓𝒎𝒂𝒏𝒄𝒆 ✈️ Careful data curation and multiple rounds of QA on annotations from human annotators ✈️Training on preference ranking improved the model performance greatly on coding and reasoning tasks 💥 𝑳𝑳𝑨𝑴𝑨3 𝑮𝒂𝒖𝒓𝒅𝒓𝒂𝒊𝒍𝒔: ✈️ Updated and new safety tools : LLAMA Gaurd2 & cybersecEval 2 ✈️ CodeShield - inference time gaurdrail for filtering insecure code ✈️ Instruction Fine-tuned model redteamed for safety : By generating adverserial prompts that try to elicit problematic responses. ✈️ LLAMAGaurd - foundational for prompt and response safety, can be easily finetuned to create a new taxonomy 💥 𝑰𝒏𝒇𝒆𝒓𝒆𝒏𝒄𝒆 & 𝑭𝒖𝒕𝒖𝒓𝒆 ✈️ Despite LLAMA3 having 1B more params than LLAMA2 7B, the improved tokenizer efficiency and GQA contribute to maintaining the inference efficiency on par with Llama 2 7B. ✈️ 400B+ parameters, Multilingual, Multimodal, and longer context window LLAMA3 would be available on AWS soon. Try it out! #generativeAI #llm #llama3 #aws #bedrock #sota

33

1 Comment

Had a great time at The Knowledge Graph Conference last week! Here are my takeaways: Not surprisingly, there was a ton of presentations and talk about GenAI and how LLMs and KGs can work together, primarily around retrieval-augmented generation (RAG). Some highlights for me: • Jon Stevens showing how they're implementing RAG in really innovative ways at AbbVie. • Thomas Smoker showing how they're doing RAG as a product at WhyHow.AI. Tom gave some insight on how constructing KGs with the goal of RAG can be different than constructing KGs for other uses.  • Jesús Barrasa from Neo4j talking about how you can combine vector based retrieval and prompt generation for what he called “contextual RAG”. • Juan Sequeda's presentation on data.world's work on benchmarking LLMs and how KGs significantly improve the accuracy of LLM results. • TopQuadrant, data.world, metaphacts GmbH, and Semantic Web Company (SWC) all announcing and demoing different RAG or LLM-assisted products But I couldn't help but wonder whether there is too much of an emphasis on LLMs right now. KGs can do a lot by themselves and introducing LLMs may improve them, but may also add risk and more inaccurate answers - what Tom Smoker referred to as 'context poisoning'. Some notes on this: • There are use cases that clearly align with LLMs+KGs, like report generation. As Brian Martin pointed out, the single biggest use case for KGs in life sciences is regulatory compliance. Brian mentioned that automating 3 or 4 reports saved them $10m! • But not all use cases require LLMs. If the questions are more about the retrieval than the generation part of RAG, you can just use a KG and ensure much higher accuracy than if you trust an LLM to interpret the results.  • Also, as Jesús Barrasa pointed out, you can do similarity in KGs WITHOUT vectors (Jaccard index for example). • Further emphasizing the power of reasoning and KGs, there was an entire panel on this topic called, 'Can Reasoning Rule in the Real World?'. For highly regulated industries, everything must follow strict rules and be fully explainable to regulators. Lily Wang from Barclays pointed out that the rules that her bank follows were the result of 100s of years of hard learning. • Also, loved hearing from Thilanka Munasinghe, Jennifer Wei, and Armin Mehrabian about how knowledge graphs are being used by NASA - National Aeronautics and Space Administration and Rensselaer Polytechnic Institute to combat climate change. • And, of course, loved to see a lot of love for SHACL throughout the conference! My colleagues Norm Friend and Holger Knublauch held a masterclass but it also came up in Tom Smoker's presentation on RAG - sometimes it is better to give an AI agent a graph that assumes a closed world than to let it think too much. #tqideas

44

Imagine this: you're facing a customer with a list of demands. Some are crystal clear, while others seem like echoes of deeper, unspoken needs. Here's the thing – we're not just order-takers. Simply nodding and executing on customer commands without a second thought can steer us toward lackluster results. It's about diving deep, reading between the lines, and aligning their wishes with the grand vision of our product. That's why last week, I crafted the Ambiguity Shooter – a GPTs that's here to bridge the gap. Just pop in what the customer shared, and voilà, it'll guide you to pinpoint those murky areas begging for a bit more clarity. Check https://lnkd.in/gC7i5htA and drop me feedback :) --- No ChatGPT Plus? Don't worry. The prompt is actually quite simple: Review the provided software requirements. Your task is to identify any ambiguities within these requirements. For each ambiguity you find, generate clear and specific questions that would help clarify these uncertainties to ensure a comprehensive understanding of the requirements.

6

2 Comments

Question for those of you ML/AI leaders who engage with UX design or research teams/vendors/contractors/colleagues: Is there 1 thing you wish UX professionals knew or understood more about ML/AI such that they could add more value to the team or the resulting products that were created? What's been hard in the past? Ex: constraints or opportunities with leveraging AI, working effectively with a data science or product team/leaders, all the data governance/quality/privacy issues that exist beyond adding an “AI feature”, or whatever.

6

GPT-4o, the next-level large language model that's shaking things up in the design world. You meticulously craft your PRD (product requirements document), outlining every feature and function. Then, with a sprinkle of AI magic, GPT-4o transforms your text into initial Figma designs! Here's why GPT-4o is a game-changer: 👉 Boost your design speed: GPT-4o can generate wireframes and layouts based on your PRD, giving you a head start and freeing up valuable time for iteration and refinement. 👉 Break the communication barrier: Ever feel like your PRDs get lost in translation between you and your developer comrades? GPT-4o can bridge the gap by using your PRD to create designs that are not only visually appealing but also technically feasible. 👉 Spark endless creativity: Let's be honest, sometimes staring at a blank screen can stifle inspiration. GPT-4o can introduce unexpected design elements based on your PRD, sparking new ideas and pushing the boundaries of your initial concept. The future of design is collaborative, efficient, and driven by powerful AI tools like GPT-4o. What are your thoughts? Share your take in the comments! Credit: Eric Vyacheslav __ 👋 Join The AI Journal community! 🚀Subscribe to the newsletter: https://lnkd.in/ex9mu2hM 🎤 Get involved in the AI conversations on X: https://lnkd.in/ew6E-ZtP #GPT4o #designrevolution #AIpowereddesign #futureofdesign #Figmamagic

4

1 Comment

Anyone interested in leveling up as a PM?? 💫 Check out Bot Mom Claire Vo on Unsolicited Feedback with Fareed Mosavat as they cover important topics like: - How to build a tool like ChatPRD - Advice for PMs if you want to build a product (even if you’re not technical) - The importance of learning SQL for data analysis & the pitfalls of shallow analysis - Why PMs should stop saying no by default - How you can use ChatPRD in your job search Watch the full interview here: https://lnkd.in/gX_7FvU8 #chatprd #clairevo #product #productmanagement 

5