I've been putting together a little SQLite database to track the top 100 songs from the iTunes RSS feed. I've built the script in Bash to do all the hard work and it's working finally, but I'm not sure if my database structure is correct, so I'm looking for some feedback on the best way to go as I am only learning SQL as I go at the moment so I don't want to dig myself into a hole when it comes to building the queries to retrieve data in time!
I have 3 tables like so;
artists_table
artist_id - PK
artist_name
songs_table
song_id - PK
artist_id - FK (from the artists table)
charts_table
chart_id - PK
song_id - FK (from the songs table)
position - (chart position 1-100)
date - (date of chart position xxxx-xx-xx)
The artists and songs table seem good to me, got foreign key constraint working...etc but I'm not sure about the charts table, anything obviously wrong with this structure?
I want to track songs/artists/positions over time so I can generate some stats...etc
Thanks,
Initial Response
I ask you about the data, in order to answer your Question, but you keep telling me about the process. No doubt, that is very important to you. And now you wish to ensure that the Record Filing System is correct.
Personally, I never write a line of code until I have the database designed. Partly because I hate to rewrite code (and I love to code). You have the sequence reversed, an unfortunate trend these days. Which means, whatever I give you, you will have to rewrite large chunks of your code.
(b.1) How exactly does it check if the artist[song] already exists ?
(b.2) How do you know that there is NOT more than occ of a specific artist/song on file ?
Right now, given the details in your Question, let's say that you have incoming, that Pussycat Dolls place 66 on the MTV chart today:
INSERT artist VALUES ( "Pussycat Dolls" ) -- succeeds, intended
INSERT artist VALUES ( "Pussycat Dolls" ) -- succeeds, unintended
INSERT artist VALUES ( "Pussycat Dolls" ) -- succeeds, unintended
Exactly which Pussycat Dolls record placed 66th today ?
When you RFS grows, and you have more fields in artist, eg. birth_date, which of the three records would you like to update ?
Ditto for Song.
How is Chart identified, is it something like US Top 40 ?
(b.1) How exactly does it check if the artist[song] already exists ?
When you execute code, it runs inside the sqLite program. What is the exact SQL string that you pass it ? Let's say you do this:
SELECT $artist_id = artist_id
FROM artist
WHERE artist_name = $artist_name
IF $artist_id = NULL
INSERT artist VALUES ( $artist_name )
Then you have going to have a few surprises when the system goes "live". Hopefully this interaction will eliminate them. Right now you have a few hundred artists.
when you have a few thousand artists, the system will slow down to a snails pace.
when things go awry, you will have duplicate artists, songs, charts.
Record Filing System
Right now, you have a pre-1970's ISAM Record Filing System, with no Relational integrity, power, or speed.
If you would like to understand more about the dangers of an RFS, in todays Relational context, please read this Answer.
Relational Database
As I understand it, you want the integrity, power, and speed of a Relational Database. Here is what you are heading towards. Obviously, it is incomplete, unconfirmed, there are may details missing, many questions remain open. But we have to model the data, only as data (as opposed to what you are going to do with it, the process), and nothing but the data.
This approach will ensure many things:
as the data grows and is added to (in terms of structure, not population), the existing data and code will not change
you will have data and referential integrity
you can obtain each of your stats via a single SELECT command.
you can execute any SELECT against the data, even SELECTs that you are not capable of dreaming about, meaning unlimited stats. As long as the data is stored in Relational form.
A database is a collection of facts about the real world, limited to the subject area of concern. Thus far we don't have facts, we have a recording of an incoming RSS stream. And the recording has no integrity, there is nothing that your code can rely on. This is heading in the direction of facts:
First Draft Music Chart TRD (Obsolete due to progression, see below.)
Response to Comments 1
Currently, I am only tracking one chart, but I see in your model that it also has the ability to track several, that is nice!
Not really. It is a side-effect of Doing Things Properly. The issue here is one of Identification. A Chart Position is not identified by RSS Feed ID, or chart_table.id, plus a PositionNo plus a DateTime. No. A Chart Position is identified as US Top 100/27 Apr 15/1… The side effect is that ChartName is part of the Identifier, and that allows multiple Charts, with no additional coding.
In these dark days of IT, people often write systems for one Country, and implement a StateCode all over the place. And then experience massive problems when they open up to an international customer base. The point is, there is no such thing as a State that does not have a Country, a State exists only in the context of a Country. So the Identifier for State must include a Country Identifier, it is (CountryCode, StateCode). Both Australia and Canada have NT for a StateCode.
If I can explain how I store the data from the rss feed, it might clear things up somewhat.
No, please. This is about the data, and only the data. Please review my previous comments on that issue, and the benefits.
I am away from my main computer at the moment, but I will respond within the next couple of hours if thats ok.
No worries. I will get to it tomorrow.
Your model does make sense to me though,
That is because you know the data values intimately, but you do not understand the data, and when someone lays it out for you, correctly, you experience pleasurable little twitches of recognition.
I don't mind having to recode everything, its a learning curve!
That's because you put the cart before the horse, and coded against data laid out in a spreadsheet, instead of designing the database first and coding against that second.
If you are not used to the Notation, please be advised that every little tick, notch, and mark, the solid vs dashed lines, the square vs round corners, means something very specific. Refer to the IDEF1X Notation.
Response to Comments 2
Just one more quick question.
Fire away, until you are completely satisfied.
In the diagram, would there be any disadvantage to putting the artist table above the song table and making the song table a child of the parent artist instead? As artists can have many songs, but each song can only have 1 artist. Is there any need for the additional table to contain just the artistPK and songPK. Could I not store the artistPK into the songs table as a FK, as a song can only exist if there is an associated artist?
Notice your attachment to the way you had it organised. I repeat:
A database is a collection of facts about the real world, limited to the subject area of concern.
Facts are logical, not physical. When those facts are organised correctly (Normalised, designed):
You can execute any SELECT against the data, even SELECTs that you are not capable of dreaming about, meaning unlimited stats. As long as the data is stored in Relational form.
When they aren't, you cant. All SQL (not only reports that are envisioned) against the data is limited to the limitations in the model, which boils down to one thing: discrete facts being recorded in logical form, or not.
With the TRD we have progressed to recording facts about the real world, limited only by the scope of the app, and not by the non-discretion of facts.
Could I not store the artistPK into the songs table as a FK, as a song can only exist if there is an associated artist?
In your working context, at this moment, that is true. But that is not true in the real world that you are recording. If the app or your scope changes, you will have to change great slabs of the db and the app. If you record the facts correctly, as they exist, not as limited to your current app scope, no such change will be necessary when the the app or your scope changes (sure, you will have to add objects and code, but not modify existing objects and code).
In the real world, Song and Artist are discrete facts, each can exist independent of the other. Your proposition is false.
Ave Maria existed for 16 centuries before Karen Carpenter recorded it.
And you already understand and accept that an Artist exists without a `Song.
Is there any need for the additional table to contain just the artistPK and songPK.
It isn't an "additional table to contain just the artistPK and songPK", it is recording a discrete fact (separate to the independent existence of Artist and Song), that a specific Artist recorded a specific Song. That is the fact that you will count on in theChartDatePosition`
Your proposition places Song as dependent on, subordinate to, Artist, and that is simply not true. Any and all stats (dreamed of or not) that are based on Song will have to navigate Artist::ArtistSong, then sort or ORDER BY, etc.
artists can have many songs, but each song can only have 1 artist.
That is half-true (true in your current working context, but not true in the real world). The truth is:
Each Artist is independent
Each Song is independent
Each Artist recorded 1-to-n Songs (via ArtistSong)
Each Song was recorded by 1-to-n Artists (via ArtistSong)
For understanding, changing your words above to form correct propositions (as opposed to stating technically correct Predicates):
Artists can have many RecordedSongs
Each RecordedSong can only have 1 Artist
Each RecordedSong can only have 1 Song
So yes, there are disadvantages, significant ones.
Which is why I state, you must divorce yourself from the app, the usage, and model the data, as data, and nothing but data.
Solution 2
I have updated the TRD.
Second Draft Music Chart TRD
Courier means example data; blue indicates a Key (Primary is always first); pipe indicates column separation; slash indicates Alternate Key (only the columns that are not in the PK are shown); green indicates non-key.
I am now giving you the Predicates. These are very important, for many reasons. The main reason here, is that it disambiguate the issues we are discussing.
If you would like more information on Predicates, visit this Answer, scroll down (way down!) to Predicate, and read that section. Also evaluate that TRD and those Predicates against it.
The index on ChartDateSong needs explanation. At first I assumed:
PK ( Chart, Date, Rank )
But then for Integrity purposes, as well as search, we need:
AK ( Chart, Date, ArtistId, SongId )
Which is a much better PK. So I switched them. We do need both. (I don't know about NONsqLite, if it has clustered indices, the AK, not the PK should be clustered.)
PK ( Chart, Date, ArtistId, SongId )
AK ( Chart, Date, Rank )
Response to Comments 3
What about the scenario when a song enters the charts with the same song_name as a record in the song_table but is completely unrelated (not a cover, completely original, but just happens to share the same name)
In civilised countries that is called fraud, obtaining benefit by deception, but I will try to think in devilish terms for a moment and answer the question.
Well, if it happens, then you have to cater for it. How does the feed inform you of such an event ? I trust it doesn't. So then your Song Identifier is still the Name.
and instead of a unique song record being created, the existing song_id is added to the artistssongs_table with the artist id, wouldn't this be a problem?
We don't know any better, so it is not a problem. No one watching that feed knows any better either. If and when you receive data informing you of that issue, through whatever channel, and you can specify it, you can change it.
Normally we have an app that allows us to navigate the hierarchies, and to change them, eg. A ReferenceMaintenance app, with an Exporer-type window on the left, and combo dialogues (list of occs on top, plus detail of one occ on the bottom) on the right .
Until then, it is not a form of corruption, because the constraint that prevents such corruption is undefined. You can't be held guilty of breaking a law that hasn't been written yet. Except in rogue states.
Although a song can have the same name, it doesn't necessarily mean it's the same record.
Yes.
Wouldn't it be better to differentiate a song by the artist?
They are differentiated by Artist.
You do appreciate that the fact of a Song, and the fact of an Artist playing a song, are two discrete facts, yes ? Please question any Predicates that do not mean perfect sense, those are the propositions that the database supports.
Ave Maria exists as an independent fact, in Song
Karen Carpenter, Celine Dion, and Yours Truly exist as three independent facts, in Artist
Karen Carpenter-Ave Maria, Celine Dion-Ave Maria, and Yours Truly-Ave Maria exist as three discrete facts in ArtistSong.
That is seven separate facts, about one Song, about three Artists.
Response to Comments 4
I do understand it now. The artistsong_table is where the 2 items "meet" and a relationship actually exists and is unique.
Yes. I just wouldn't state it in that way. The term Fact has a technically precise meaning, over and above the English meaning.
A database is a collection of facts about the real world, limited to the subject area of concern.
Perhaps read my Response 3 again, with that understanding of Fact in mind.
Each ArtistSong row is a Fact. That depends on the Fact of an Artist, and the Fact of a Song. It establishes the Fact that that Artist recorded that Song. And that ArtistSong Fact is one that other Facts, lower in the hierarchy, will depend upon.
"Relationship ... actually". I think you mean "instance". The relationship exists between the tables, because I drew a line, and you will implement a Foreign Key Constraint. Perhaps think of Fact as an "instance".
Just to make sure I understand the idea correctly, if I were to add "Genre" into the mix, would I be correct in thinking that a new 'independent' table genre_table would be created and the artistsong_table would inherit its PK as an FK?
Yes. It is a classic Reference or Lookup table, the Relationship will be Non-identifying. I don't know enough about the music brothelry to make any declarations, but as I understand it, Genre applies to a Song; an Artist; and an ArtistSong (they can play a Song in a Genre that is different to the Song.Genre). You have given me one, so I will model that.
The consequence of that is, when you are inserting rows in ArtistSong, you will have to have the Genre. If that is in the feed, well and good, if not, you have a processing issue to deal with. The simple method to overcome that is, implement a Genre "", which indicates to you that you need to determine it from other channels.
It is easy enough to add a classifier (eg. Genre) later, because it is a Non-identifying Relationship. But Identifying items are difficult to add later, because they force the Keys to change. Refer para 3 under my Response 1.
You are probably ready for a Data Model:
Third Draft Music Chart Data Model
It all depends on the relationships (one-to-one, one-to-many, many-to-many) your data is going to have.
The way you implemented your charts table indicates that:
Each chart has only/belongs to one song
A song can have many charts
It is a one-to-many relationship. And if that was what you intended then everything seems fine.
However:
If your charts can have many songs and a song will have only one
chart (also a one-to-many relationship but reversed), the song_id column needs to
be taken out from the charts table and the songs table needs
chart_id column in.
If your charts can have many songs and your songs can have many charts as well (many-to-many relationship), then you need a "joint table" which could be something like this:
TABLE: charts_songs, COLUMNS: id, chart_id, song_id, position
I need some help in designing car booking application.
There is a document with information about car (title, model, brand, info, etc.)
Problems I'm stuck with are:
How to store available booking days? (I suppose I could use nested
free date range objects in array)
How to store price per day (it's possible to have individual price
per day)?
Booking days and prices could change often. So the third question is: "how to update them cleverly (partially), so I shouldn't read the document, and then store it". I'm looking at script solution using
update api (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-update.html), but it looks ugly. Maybe there are other approaches?
Thanks,
Alex
with the introduction of the range datatypes, there is no need to use a real nested object, if you meant that.
That might also help you with storing the prices, but that could just be any object I suppose (it depends if you want to search for that as well).
Update API was made for exactly that use-case, that you do not need to get the whole document, so that shounds like a plan.
I know that it has been told almost anything related to time table modeling in RDBS, but I can not find any well written documentation about available techniques to store time tables in DB.
My case:
I have table which holds available places, and table with actual classes.
Each place has it's own unique schedule
Each class can be scheduled in any place, and any time, with few exceptions:
One class can take one time-slot (Example: If class A is scheduled in place P1 at 12:00 for 1hour duration, next occurrence of class A can only be placed before 12:00 or after 13:00, in any place, which has free time-slot; It's forbidden to schedule class A in one time in two places)
In one place it can be one class with time-slot
Model should support versioning/history of scheduled classes
Now, how I can represent this data model in an SQL DB?
I'm not looking ready-to-use exact schema, rather I will be glad if anyone can write available modelling techniques and their comparison, which I can use to solve this task
For example: For tree-structure/hierarchical data, there is well documented "modified preorder tree traversal algorithm", is there some similar algorithm/technique to deal with time-slots?
A timetable is a matrix. Down the left hand side we have LOCATIONS. Across the top we have TIMESLOTS. The intersection of any given permutation of LOCATION and TIMESLOT is a cell with either a CLASS or null.
To model this we need a table (entity) of LOCATIONS, which is pretty fixed data. We need a table of TIMESLOTS (date/times) which is ever growing. We need a table CLASSES, which is also pretty fixed. Finally we need an intersection table CLASS_TIMESLOT_LOCATIONS. This is where the magic happens. This table has three foreign keys, one to CLASSES, one to LOCATIONS, one to TIMESLOTS. Its primary key is (LOCATION_ID, TIMESLOT_ID) but it also needs a unique constraint on (CLASS_ID, TIMESLOT_ID).
You are asking a modelling question, but there are a couple of implementation details which you will need to think about. They won't chnage the logical model but they will affect how you work with the physical tables. The first consideration is whether to spawn all the potential TIMESLOTS, and, if so, how big a window you store. The second is whether to store null entries for the intersection table, CLASS_TIMESLOT_LOCATIONS.
There are no straightforward answers here: some database products will find it easier to "fill in the gaps" than others. Also, generating the absent records on the fly may be too much of a performance hit, in which case disk space is a good trade-off.
As for storing history, this is presumably for storing changes to the schedule. Use separate tables for this, populated by triggers (you could use stored procedures but triggers is the industry standard). Don't be tempted to store history in the main tables. It breaks the normalised model and causes all sorts of grief.
From what I see in your question it looks like you have several constraints you would like to have handled on the database side.
• I have table which holds available places, and table with actual classes.
More can be elaborated on the table design, but this just needs a table schema to hold the information you need
• Each place has it's own unique schedule
How about creating a trigger on inserts to make sure that the class that is being inserted into the schedule does not conflict with any other schedule?
• Each class can be scheduled in any place, and any time, with few exceptions:
• One class can take one time-slot (Example: If class A is scheduled in place P1 at 12:00 for 1hour duration, next occurrence of class A can only be placed before 12:00 or after 13:00, in any place, witch has free time-slot; It's forbidden to schedule class A in one time in two places)
I would handle this constraint in a trigger also
• In one place it can be one class with time-slot
Have this constraint handled in a trigger
• Model should support versioning/history of scheduled classes
Have a separate table which mirrors the actual table that you have for schedules. As new records get inserted into the main table, you can trigger the updates and times for the updates/inserts/deletes into the table which has the history
Hope this helps you with some ideas.
-Vijay