Interview ｜ A detailed explanation of ＂China Audio-visual Big Data＂
It has been nearly three years since the system was put into trial operation, and the big data system for comprehensive evaluation of radio and television program ratings of the General Administration of China has had a profound impact on the industry. What is the core architecture of the system? How does the system highlight innovation and key applications? What is the focus of the next step? Recently, the reporter of "Radio and Television Review" had an exclusive interview with the person in charge of the Big Data Research Center for Program Comprehensive Evaluation of the Radio and Television Planning Institute of the General Administration.
Author: Jiang Yong
"Buzz."
Walking into the communication network computer room of the big data system for comprehensive evaluation of program ratings of the General Administration of China, the roaring air conditioning sound made the recording pen stop working.
Communication network room of big data system for comprehensive evaluation of program ratings of General Administration of China.
On the rows of racks, hundreds of servers, switches and routers are running at high speed, and countless data are collected, stored and calculated here in binary form, which eventually become a series of professional numbers on the viewing table.
The person in charge of the Big Data Research Center for Program Comprehensive Evaluation said: "The viewing data of more than 300 million users (cable TV, IPTV, Internet TV) in China are gathered here, which is equivalent to the’ brain’ and’ warehouse’ of the whole system."
On December 26, 2018, the big data system for comprehensive evaluation of radio and television program ratings of the General Administration of China was put into trial operation. In the past three years, relying on the real and pollution-free viewing statistics and analysis, the system has provided important support for public opinion guidance, program broadcast management, etc., and has also become the main reference for daily assessment and advertising investment promotion of TV stations and other units.
"National team data", "pollution-free data" and "dehydration data" are the basic cognition of the industry on the big data system for comprehensive evaluation of ratings, but there are not many in-depth issues about how to create, how to operate and what innovative features this system has. Recently, the reporter of "Radio and Television Review" conducted an exclusive interview with the person in charge of the Big Data Research Center for Program Comprehensive Evaluation of the Radio and Television Planning Institute of the General Administration, revealing the origin, operation and development of this system.
On-line: an important measure to control viewing fraud from the source
The first appearance of the big data system for comprehensive evaluation of program ratings was at the end of 2018. However, as early as 2016, under the leadership of the State Administration of Radio, Film and Television, the Radio and Television Planning Institute took the lead in organizing a number of radio and television stations, network operators, research institutes, universities and enterprises, and successively carried out research and development of core technologies for viewing, standard formulation and small-scale experiments.
The big data system for comprehensive evaluation of radio and television program ratings of the General Administration of China was put into trial operation.
The person in charge said that for a long time, the CPC Central Committee has attached great importance to the construction of the viewing survey system and demanded the establishment of an authoritative and convincing new viewing survey system to promote the high-quality and innovative development of radio and television. The Party Group of the General Administration has conscientiously implemented the central government’s decision-making and deployment, and taken the construction of a comprehensive evaluation big data system for program viewing as an important measure to implement the spirit of the central government.
At the same time, the authoritative comprehensive evaluation of program ratings is also the general trend of the industry. All sectors of society call for the establishment and comprehensive application of a new audience survey system as soon as possible, so as to solve the outstanding problems in the field of audience survey from the source.
In this context, the big data system for comprehensive evaluation of program ratings came into being.
Attack: complete the system construction in three stages
From 0 to 1, innovative construction.
The big data system for comprehensive evaluation of program viewing abandons the traditional viewing survey mode and innovatively builds a new big data viewing survey system with cloud computing, whole network, full sample and big data as its important features.
The person in charge bluntly said, "It is very difficult, from data access to system construction, model and index design to system online, all from scratch".
Since 2016, the construction process of the system has been divided into three stages with different periods and different key points.
In the first stage, the Radio and Television Planning Institute led relevant units to carry out independent research and development of the core technology of viewing survey, formulated and published a number of industry specifications such as viewing data element set, exchange interface and cleaning rules, and independently mastered a number of core technology patents and software copyrights.
At the same time, by studying the theory of big data viewing survey, the index system and calculation method are determined to solve the problems of sample analysis, statistical rules and processing methods, data modeling and analysis of big data viewing under the new situation.
In the second stage, the Planning Institute organized a number of cable TV networks, IPTV and Internet TV institutions to carry out tens of millions of technical experiments, which verified the rationality and feasibility of the basic theory, technical scheme, index system, survey standards, model algorithms, quality control, security and privacy protection strategies of the large sample viewing survey formed in the first stage.
At this stage, the experiment was highly praised by experts, and it was agreed that "it can meet the needs of ultra-large-scale, multi-source heterogeneous viewing data analysis and comprehensive evaluation of programs".
In the third stage, in the second half of 2018, the State Administration of Radio, Film and Television decided to carry out the construction of "Big Data System for Comprehensive Evaluation of Radio and Television Program Audience" by the Planning Institute on the basis of previous scale experiments and trial and error.
"From project research to experimental verification to system construction and operation, it embodies the efforts and efforts of many staff." The person in charge said.
As far as the industry is concerned, the completion and operation of the system has played an important role in establishing a scientific, true and effective ratings evaluation system and fundamentally solving the problem of ratings fraud.
Architecture: five characteristics to strengthen service management
How does the big data system for comprehensive evaluation of program ratings work?
The person in charge introduced that the process of viewing big data involves the collection, cleaning, warehousing, analysis and application of viewing big data. "The whole process is a bit like cooking."
The operation flow of the system is that the user’s viewing data is directly collected from the set-top box by operators (such as gehuayouxian) and transmitted to the viewing comprehensive evaluation big data system through a secure channel. After collecting, gathering, cleaning and converting massive data, the system carries out modeling, statistics, analysis and other work, and outputs big data viewing survey indicators.
There is no human intervention in the whole process. At the critical data acquisition stage, the system is equipped with a three-level verification mechanism of normative verification, integrity verification and rationality verification, which effectively prevents data from being contaminated.
On the whole, the innovation of big data system can be summarized into five aspects:
First, the sample is complete, the coverage is wide, and there are massive data sources. The system has realized the aggregation and analysis of the viewing data of 300 million cable TV, IPTV and Internet TV users nationwide, covering various viewing modes such as live broadcast, review and on-demand.
The second is big data, cloud computing, efficient processing and accurate home. Based on big data and cloud computing technology, the system can efficiently and timely count the super-large-scale viewing data and analyze the accurate home-to-home, which can not only reflect the viewing situation of popular programs and prime time, but also accurately capture the viewing characteristics of minority programs and marginal time.
The third is to prevent manipulation and pollution, and fundamentally solve the falsification of viewing. The data collection, cleaning, analysis and presentation of the system are seamlessly connected, and the whole process is automated and closed to prevent human manipulation. The system is based on massive data statistics, and the influence of individual sample data pollution on statistical results can be ignored, so the system has strong anti-pollution ability.
Fourth, multi-dimensional and all-round, and integrated analysis leads development. The system innovatively established an indicator system of viewing big data covering more than 80 core indicators in eight aspects. Through deep mining and timely feedback of viewing data, it guided content selection, material integration, demand combination, analysis and prediction, creation and production, changed the traditional program production mode, and effectively guided the healthy development of the industry.
The fifth is all-media, openness, and presuppose a new orientation in the future. The system adapts to the development of media convergence and the new changes of communication pattern and communication environment, continuously enriches the sources of TV viewing data, and will comprehensively cover different communication channels such as cable TV, satellite live broadcast, IPTV, Internet TV and network audio-visual fields, and preset the new positioning and model under the general trend of national cable TV network integration and 5G mobile application in advance.
Impact: promoting industry development with data
At the beginning of the trial operation of the big data system for comprehensive evaluation of program ratings, many media commented that "the problem of industry ratings was rectified in a radical way".
Up to now, the positive impact of the system on the industry has already appeared.
According to the person in charge, at present, the system has exported more than 30,000 professional data analysis reports to the central leadership, Publicity Department of the Communist Party of China, industry authorities, national television stations, etc., supporting publicity and regulation, and helping TV stations to create, produce and broadcast management, which has been affirmed by many parties.
Through data query and analysis or data report presentation, it provides comprehensive data support for CCTV, China Education TV, movie channels and national satellite TV channels, and becomes the main data for daily program assessment.
On December 17, 2019, the Program Comprehensive Evaluation Big Data Research Center officially released the viewing data to the public under the brand of "China Audio-Visual Big Data (CVB)". After the release of CVB data, it was well received by the public and applauded the SARFT’s "combination boxing" and "heavy punching" to control ratings fraud.
The first viewing data released by "China Audio-Visual Big Data (CVB)" to the public.
At the end of 2020, formulated by the State Administration of Radio and Television and approved by the National Bureau of Statistics, the "Statistical Survey System of Radio and Television Program Audience Big Data" was released, and the radio and television industry became the first industry to conduct statistical surveys of government departments in the form of big data.
Talking about the focus of the next step, the person in charge introduced that there are about 50 employees in the Big Data Research Center for Program Comprehensive Evaluation, who are responsible for data access and analysis, data report compilation and release, public opinion analysis, data application promotion and so on.
On the basis of the existing work, the Planning Institute will rely on large system resources to promote the implementation of the "Big Data System Expansion for Comprehensive Evaluation of Radio and Television Program Audience" project, improve the scale of system data aggregation, optimize the data chain mechanism, improve the quality of data throughout the life cycle, ensure the accuracy, completeness and timeliness of data collection and aggregation, and ensure the authenticity, objectivity and order of data release and application.
With the multi-screen viewing environment, the Planning Institute will accelerate the data access of Internet TV and Internet audio-visual websites, build a data index system for the network audio-visual field, realize the effective aggregation and analysis of network audio-visual viewing data, and give full play to the role of "China audio-visual big data".
A few days ago, the State Administration of Radio and Television issued the Notice on Further Strengthening the Management of Cultural Programs and Their Personnel, which clearly stated: "Look at quantitative indicators such as ratings and click-through rates scientifically and increase the promotion and application of’ China Audio-visual Big Data’."
Regarding the new requirements, the person in charge said, "We will continue to optimize and improve the big data system for comprehensive evaluation of program viewing, better serve publicity management, support the development of the industry, and achieve comprehensive and diversified data support for the dissemination of quality programs."
(Ning Yahong also contributed to this article)
Editor | Ning Yahong Sui Fangfang