Conversation data collection and cleaning is a crucial step in developing a chatbot that can have natural human-like conversations. To collect high quality data, it is important to plan the data collection process carefully.
The first step would be to define clear goals and guidelines for the type and content of conversations needed for training. This will help determine what domains or topics the conversations should cover, what types of questions or statements the chatbot should be able to understand and respond to, and at what level of complexity. It is also important to outline any sensitive topics or content that should be excluded from the training data.
With the goals defined, I would work to recruit a group of diverse conversation participants. To collect natural conversations, it is best if the participants do not know they are contributing to a chatbot training dataset. The participants should represent different demographics like age, gender, location, personality types, interests etc. This will help collect conversations covering varied perspectives and styles of communication. At least 500 participants would be needed for an initial dataset.
Participants would be asked to have text-based conversations using a custom chat interface I would develop. The interface would log all the conversations anonymously while also collecting basic metadata like timestamps, participant IDs and word counts. Participants would be briefed that the purpose is to have casual everyday conversations about general topics of their choice.
Multiple conversation collection sessions would be scheduled at different times of the day and week to account for variability in communication styles based on factors like time, mood, availability etc. Each session would involve small groups of 3-5 participants conversing freely without imposed topics or structure.
To encourage natural conversations, no instructions or guidelines would be provided on the conversation content or style during the sessions. Participants would be monitored and prompted to continue conversations that seem to have stalled or moved to restricted topics. The logging interface would automatically end sessions after 30 minutes.
Overall, I aim to collect at least 500 hours of raw conversational text data through these participant sessions, spread over 6 months. The collected data would then need to be cleaned and filtered before use in training.
For data cleaning, I would develop a multi-step pipeline involving both automated tools and manual review processes. First, all personally identifiable information like names, email IDs, phone numbers would be removed from the texts using regex patterns and string replacements. Conversation snippets with significantly higher word counts than average, possibly due to copy-paste content would also be filtered out.
Automated language detection would be used to remove any non-English conversations from the multilingual dataset. Text normalization techniques would be applied to handle issues like spelling errors, slang words, emojis etc. Conversations with prohibited content involving hate speech, graphic details, legal/policy violations etc would be identified using pretrained classification models and manually reviewed for removal.
Statistical metrics like total word counts, average response lengths, word diversity would be analyzed to detect potentially problematic data patterns needing further scrutiny. For example, conversations between the same pair of participants occurring too frequently within short intervals may indicate lack of diversity or coaching.
A team of human annotators would then manually analyze a statistically significant sample from the cleaned data, looking at aspects like conversation coherence, context appropriateness of responses, naturalness of word usage and style. Any remaining issues not caught in automated processing like off-topic, redundant or inappropriate responses would be flagged for removal. Feedbacks from annotators would also help tune the filtering rules for future cleanup cycles.
The cleaned dataset would contain only high quality, anonymized conversation snippets between diverse participants, sufficient to train initial conversational models. A repository would be created to store this cleaned data along with annotations in a structured format. 20% of the data would be set aside for evaluation purposes and not used in initial model training.
Continuous data collection would happen in parallel to model training and evaluation, with each new collection undergoing the same stringent cleaning process. Periodic reviews involving annotators and subject experts would analyze any new issues observed and help refine the data pipeline over time.
By planning the data collection and cleaning procedures carefully with clearly defined goals, metrics for analysis and multiple quality checks, it aims to develop a large, diverse and richly annotated conversational dataset. This comprehensive approach would help train chatbots capable of nuanced, contextual and ethically compliant conversations with humans.