About the challenge
GitHub Agentic Workflows (GH-AW) represent an emerging form of repository automation in which developers define agentic workflows in Markdown and execute them through GitHub Actions. These workflows combine natural-language instructions with structured configuration, allowing AI agents to interact with repositories and perform software engineering tasks.
Despite the growing interest in agentic software engineering, little is known about how developers specify, configure, and evolve these workflows in practice.
The GHAW-H dataset provides an empirical resource for investigating these questions. It covers 262 public GitHub repositories that used GH-AW between February 13 and June 6, 2026, and connects agentic workflow specifications with their compiled representations and version histories. The dataset includes natural-language instructions written in Markdown, structured frontmatter declarations, compiled workflow definitions in YAML, and historical versions of these artifacts.
Research opportunities
GHAW-H enables researchers to investigate how agentic workflows are specified and evolve in real-world software repositories.
The dataset supports a broad spectrum of empirical software engineering research methods. Participants are free to define their own study designs, methods, tools, and experimental protocols according to the research questions they investigate.
Example research directions include, but are not limited to:
- Workflow specifications: How do developers express tasks and goals in natural language when defining agentic workflows?
- Frontmatter configuration: How are GH-AW capabilities, permissions, triggers, tools, and execution options configured in practice?
- Specification structure: What characteristics distinguish different types of agentic workflow specifications?
- Specification–compilation relationships: How are Markdown specifications reflected in their compiled workflow definitions?
- Workflow evolution: How do natural-language instructions, frontmatter configurations, and compiled workflows change over time?
- Co-evolution: How do changes to natural-language instructions relate to changes in structured configuration and compiled workflow definitions?
- Comparison: How do GH-AW specifications differ from other forms of agent or software automation configuration?
- Methods and tools: How can NLP, machine learning, LLM-based, or other analysis techniques support the study, understanding, or development of agentic workflows?
These questions are intended as examples rather than an exhaustive research agenda. Researchers are encouraged to formulate additional questions enabled by GHAW-H and, when appropriate, to enrich the dataset with additional publicly available information.
Different types of contributions are welcome, including empirical studies, tool-oriented papers, comparative analyses with related datasets, and replication or extension studies.
How to participate
First, familiarize yourself with GHAW-H and its artifacts:
Zenodo: link to be added.
Use GHAW-H to investigate a research question related to natural language and agentic workflows in software engineering. Researchers may use the dataset as provided or combine it with additional data when this is relevant to their study.
The choice of research questions and methodological approach is deliberately open. Studies may employ quantitative, qualitative, or mixed empirical methods, as well as NLP, machine learning, LLM-based analysis, mining software repositories, manual analysis, or other approaches appropriate to the research question.
Submissions
A GitHub Agentic Workflows Challenge paper should clearly describe:
- the software engineering problem or research question being investigated and its relevance to Natural Language-based Software Engineering;
- the role of natural language in the investigated problem and the GHAW-H artifacts used in the study;
- the study design, methods, and tools employed;
- the main findings or resulting contribution;
- the implications of the results for the use, analysis, or development of natural language-based software engineering artifacts and tools; and
- the conclusions of the work.
Submissions should clearly identify the version of GHAW-H used to support reproducibility. Authors who augment GHAW-H with additional data should document the corresponding data sources and collection or extraction procedures.
We encourage authors to make analysis code, derived data, tools, and other research artifacts available whenever possible to facilitate reproducibility and further research.