US5828809A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for extracting indexing information from digital video data
Assignee: Panasonic Holdings Corp
•
Filing: 1996-10-01
•
Granted: 1998-10-27
•
Expires: 2016-10-01
Abstract
Abstract A method and apparatus to automatically index the locations of specified events on a video tape. The events, for example, include touchdowns, fumbles and other football-related events. An index to the locations where these events occur are created by using both speech detection and video analysis algorithms. A speech detection algorithm locates specific words in the audio portion data of the video tape. Locations where the specific words are found are passed to the video analysis algorithm. A range around each of the locations is established. Each range is segmented into shots using a histogram technique. The video analysis algorithm analyzes each segmented range for certain video features using line extraction techniques to identify the event. The final product of the video analysis is a set of pointers (or indexes) to the locations of the events in the video tape.
Inventors
- 👤 Yuh-Lin Chang
- 👤 Wenjun Zeng
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-23 19:35
CPC Classifications
H; H04; H04N; H04N17/00; H04N17/004; G; G06; G06F; G06F16/00; G06F16/70; G06F16/71; G06F16/78; G06F16/783; G06F16/7834; G06F16/7844; G06F16/7847; G06F16/785; G06V; G06V20/00; G06V20/40; G11; G11B; G11B27/00; G11B27/10; G11B27/19; G11B27/28; H04H; H04H60/00; H04H60/35; H04H60/37; H04H60/48; H04H60/56; H04H60/58; H04H60/59; G10; G10L; G10L15/00; G10L15/26; G11B2220/00; G11B2220/90; H04N5/00; H04N5/14; H04N5/147
Claims Analysis
Total Claims: 52
Claim 1
Independent Claim
191 words
A computer-implemented speech and video analysis system for creating an index to indicate locations of a first event occurring within audio-video data, said audio-video data containing audio data synchronized with video data to represent a plurality of events, said first event having at least one audio-feature and at least one video-feature indicative of said first event, comprising the steps of: (a) providing a model speech database for storing speech models representative of said audio-feature; (b) providing a model video database for storing video models representative of said video-feature; (c) performing wordspotting to determine candidates by comparing said audio data with said stored speech models, said candidates indicating positions of said audio-feature within said audio data; (d) establishing predetermined ranges around each of said candidates; (e) segmenting into shots those portions of said video data which are located within said ranges; (f) analyzing said segmented video data to determine video-locations based on a comparison between said segmented video data and said stored video models, said video-locations indicating positions of said video-feature within said segmented video data; and (g) generating an index to indicate locations of said first event based on said video-locations.
Claim 2
Dependent Claim
191 words
A computer-implemented speech and video analysis system for creating an index to indicate locations of a first event occurring within audio-video data, said audio-video data containing audio data synchronized with video data to represent a plurality of events, said first event having at least one audio-feature and at least one video-feature indicative of said first event, comprising the steps of: (a) providing a model speech database for storing speech models representative of said audio-feature; (b) providing a model video database for storing video models representative of said video-feature; (c) performing wordspotting to determine candidates by comparing said audio data with said stored speech models, said candidates indicating positions of said audio-feature within said audio data; (d) establishing predetermined ranges around each of said candidates; (e) segmenting into shots those portions of said video data which are located within said ranges; (f) analyzing said segmented video data to determine video-locations based on a comparison between said segmented video data and said stored video models, said video-locations indicating positions of said video-feature within said segmented video data; and (g) generating an index to indicate locations of said first event based on said video-locations.
Claim 3
Dependent Claim
34 words
The method according to claim 1 wherein said predetermined ranges have a starting position of one minute before each of said candidates and an ending position of two minutes after each of said candidates.
Claim 4
Dependent Claim
19 words
The method according to claim 1 further comprising the step of deriving said audio-video data from a video tape.
Claim 5
Dependent Claim
15 words
The method according to claim 1 further comprising the step of digitizing said audio data.
Claim 6
Dependent Claim
15 words
The method according to claim 1 further comprising the step of digitizing said video data.
Claim 7
Dependent Claim
13 words
The method according to claim 1 wherein said audio-feature is a predefined utterance.
Claim 8
Dependent Claim
26 words
The method according to claim 6 further comprising the steps of: determining energy of said predefined utterance; and storing said determined energy in said speech models.
Claim 9
Dependent Claim
30 words
The method according to claim 7 further comprising the step of determining said candidates based on Euclidean distance between said energy of said audio data and said energy speech models.
Claim 10
Dependent Claim
29 words
The method according to claim 6 further comprising the steps of: determining Hidden Markov Models of said predefined utterance; storing said determined Hidden Markov Models in said speech models.
Claim 11
Dependent Claim
32 words
The method according to claim 9 further comprising the step of determining said candidates based on a Hidden Markov Model comparison between said audio data and said Hidden Markov Model speech models.
Claim 12
Dependent Claim
29 words
The method according to claim 6 further comprising the steps of: determining a phonetic model of said predefined utterance; and storing said determined phonetic model in said speech models.
Claim 13
Dependent Claim
30 words
The method according to claim 11 further comprising the step of determining said candidates based on a dynamic time warping analysis performed between said audio data and said speech models.
Claim 14
Dependent Claim
25 words
The method according to claim 1 wherein each of said shots being a contiguous set of video data depicting a discrete activity within an event.
Claim 15
Dependent Claim
33 words
The method according to claim 13 further comprising the step of segmenting said video data based upon a histogram difference X 2 comparison between said segmented video data and said stored video models.
Claim 16
Dependent Claim
22 words
The method according to claim 13 further comprising the step of storing line representations of said video-feature within said stored video models.
Claim 17
Dependent Claim
19 words
The method according to claim 15 further comprising the step of performing line extraction upon said segmented video data.
Claim 18
Dependent Claim
22 words
The method according to claim 14 further comprising the step of storing color characteristics of said video-feature within said stored video models.
Claim 19
Dependent Claim
31 words
The method according to claim 17 further comprising the step of determining video-locations based on comparing color data of said video data with said color characteristics of said stored video models.
Claim 20
Dependent Claim
23 words
The method according to claim 13 further comprising the step of storing texture characteristics of said video feature within said stored video models.
Claim 21
Dependent Claim
31 words
The method according to claim 19 further comprising the step of determining video-locations based on comparing texture data of said video data with said texture characteristics of said stored video models.
Claim 22
Dependent Claim
39 words
The method according to claim 1 further comprising the step of storing a predefined transition of shots within said video models, each of said shots being a contiguous set of video data depicting a discrete activity within an event.
Claim 23
Dependent Claim
20 words
The method according to claim 21 wherein said discrete activity includes two football teams lining up in a football formation.
Claim 24
Dependent Claim
18 words
The method according to claim 21 wherein said discrete activity includes a football team attempting a field goal.
Claim 25
Dependent Claim
28 words
The method according to claim 21 wherein said predefined transition of shots includes a lining up shot, an action shot, an aftermath shot, and an extra point shot.
Claim 26
Dependent Claim
30 words
The method according to claim 21 further comprising the step of comparing said shots from said video data to said stored predefined transition of shots to identify said first event.
Claim 27
Independent Claim
218 words
An apparatus for creating an index to indicate locations of a first event occurring within audio-video data, said audio-video data containing audio data synchronized with video data to represent a plurality of events, said first event having at least one audio-feature and at least one video-feature indicative of said first event, comprising: a model speech database for storing speech models representative of said audio-feature; a model video database for storing video models representative of said video-feature; a wordspotter coupled to said model speech database for determining candidates based on comparison between said audio data with said stored speech models, said candidates indicating positions of said audio-feature within said audio data; range establishing means coupled to said wordspotter for establishing predetermined ranges around each of said candidates; a segmenting device coupled to said range establishing means for segmenting into shots those portions of said video data which are located within said ranges; an video analyzer coupled to said segmenting device and to said model video database for determining video-locations based on a comparison between said video data and said stored video models, said video-locations indicating positions of said video-feature within said video data; and an indexer coupled to said video analyzer for creating indicating the locations of said first event within said audio-video data based on said determined video-locations.
Claim 28
Independent Claim
218 words
An apparatus for creating an index to indicate locations of a first event occurring within audio-video data, said audio-video data containing audio data synchronized with video data to represent a plurality of events, said first event having at least one audio-feature and at least one video-feature indicative of said first event, comprising: a model speech database for storing speech models representative of said audio-feature; a model video database for storing video models representative of said video-feature; a wordspotter coupled to said model speech database for determining candidates based on comparison between said audio data with said stored speech models, said candidates indicating positions of said audio-feature within said audio data; range establishing means coupled to said wordspotter for establishing predetermined ranges around each of said candidates; a segmenting device coupled to said range establishing means for segmenting into shots those portions of said video data which are located within said ranges; an video analyzer coupled to said segmenting device and to said model video database for determining video-locations based on a comparison between said video data and said stored video models, said video-locations indicating positions of said video-feature within said video data; and an indexer coupled to said video analyzer for creating indicating the locations of said first event within said audio-video data based on said determined video-locations.
Claim 29
Dependent Claim
34 words
The apparatus according to claim 26 wherein said predetermined ranges have a starting position of one minute before each of said candidates and an ending position of two minutes after each of said candidates.
Claim 30
Dependent Claim
16 words
The apparatus according to claim 26 wherein said audio-video data is derived from a video tape.
Claim 31
Dependent Claim
14 words
The apparatus according to claim 26 wherein said audio data is digital audio data.
Claim 32
Dependent Claim
14 words
The apparatus according to claim 26 wherein said video data is digital video data.
Claim 33
Dependent Claim
13 words
The apparatus according to claim 26 wherein said audio-feature is a predefined utterance.
Claim 34
Dependent Claim
18 words
The apparatus according to claim 31 wherein said speech models are based upon energy of said predefined utterance.
Claim 35
Dependent Claim
27 words
The apparatus according to claim 32 wherein said wordspotter selects said audio-locations based on Euclidean distance between said energy of said audio data and said speech models.
Claim 36
Dependent Claim
21 words
The apparatus according to claim 31 wherein said speech models are based upon Hidden Markov speech models of said predefined utterance.
Claim 37
Dependent Claim
29 words
The apparatus according to claim 34 wherein said wordspotter selects said audio-locations based on a Hidden Markov Model comparison between said audio data and said Hidden Markov speech models.
Claim 38
Dependent Claim
20 words
The apparatus according to claim 31 wherein said speech models are based upon a phonetic model of said predefined utterance.
Claim 39
Dependent Claim
28 words
The apparatus according to claim 36 wherein said wordspotter selects said audio-locations based on a dynamic time warping analysis performed on said audio data and said speech models.
Claim 40
Dependent Claim
25 words
The method according to claim 26 wherein each of said shots being a contiguous set of video data depicting a discrete activity within an event.
Claim 41
Dependent Claim
25 words
The apparatus according to claim 38 wherein said segmenter device segments said portions of said video data based upon a histogram difference X 2 comparison.
Claim 42
Dependent Claim
18 words
The apparatus according to claim 38 wherein said video models are based upon line representations of said video-feature.
Claim 43
Dependent Claim
25 words
The apparatus according to claim 40 wherein said video analyzer includes a line extraction device for representing said video data as a set of lines.
Claim 44
Dependent Claim
16 words
The apparatus according to claim 38 wherein said video models include color characteristics of said video-feature.
Claim 45
Dependent Claim
31 words
The apparatus according to claim 42 wherein said video analyzer includes a color analysis device for comparing color data of said video data with said color characteristics of said video models.
Claim 46
Dependent Claim
16 words
The apparatus according to claim 38 wherein said video models include texture characteristics of said video-feature.
Claim 47
Dependent Claim
31 words
The apparatus according to claim 44 wherein said video analyzer includes a texture analysis device for comparing texture data of said video data with said texture characteristics of said video models.
Claim 48
Dependent Claim
36 words
The apparatus according to claim 26 wherein said video models are based upon a predefined transition of shots, each of said shots being a contiguous set of video data depicting a discrete activity within an event.
Claim 49
Dependent Claim
20 words
The apparatus according to claim 46 wherein said discrete activity includes two football teams lining up in a football formation.
Claim 50
Dependent Claim
18 words
The apparatus according to claim 46 wherein said discrete activity includes a football team attempting a field goal.
Claim 51
Dependent Claim
28 words
The apparatus according to claim 46 wherein said predefined transition of shots includes a lining up shot, an action shot, an aftermath shot, and an extra point shot.
Claim 52
Dependent Claim
27 words
The apparatus according to claim 46 wherein said video analyzer compares shots from said video data to said predefined transition of shots to identify said first event.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2001-10-26 | 2002-04-27 | 2002-10-27 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2005-10-26 | 2006-04-27 | 2006-10-27 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2009-10-26 | 2010-04-27 | 2010-10-27 | $7700.00 | Window Closed / Lapsed |