Changes for page Development-Database vs Filesystem
Last modified by Pascal Robert on 2011/02/22 10:35
From version 2.1
edited by smmccraw
on 2007/07/08 09:45
on 2007/07/08 09:45
Change comment:
There is no comment for this version
To version 5.1
edited by Pascal Robert
on 2010/09/13 00:27
on 2010/09/13 00:27
Change comment:
There is no comment for this version
Summary
-
Page properties (3 modified, 0 added, 0 removed)
Details
- Page properties
-
- Title
-
... ... @@ -1,1 +1,1 @@ 1 - Programming__WebObjects-Web Applications-Development-Database vs Filesystem1 +Development-Database vs Filesystem - Author
-
... ... @@ -1,1 +1,1 @@ 1 -XWiki. smmccraw1 +XWiki.probert - Content
-
... ... @@ -1,20 +1,20 @@ 1 -== Overview 1 +== Overview == 2 2 3 3 There is an ongoing debate, generally related to media files, about whether to store media in the database or whether to store media on the fileystem and just store the reference in the database. 4 4 5 5 This article attempts to track some of the notable writings about the debate. 6 6 7 -== Joe Moreno 7 +== Joe Moreno == 8 8 9 9 Keep in mind that the purpose of a database is to store data to be search and retrieved.<BR>It would be a rare case when you'd actually send a query to a database that consisted of an image blob (i.e. search for an image that matches certain binary data). More than likely, you'd perform a search for an image based on its meta-data like date, time, image name, or file system path. A good solution is to store the path to the medium and then simply build the reference URL for the client's browser to reference or have the application retrieve the medium from the file system and serve it up through the WebObjects adaptor. In the former case you can keep the media under the Web server (say image thumbnails) and, in the latter case, you can keep full size images anywhere else on the server's file system and server them up based on a user's profile (i.e. did they successfully check out?, etc). 10 10 11 -== Michael Engelhart 11 +== Michael Engelhart == 12 12 13 13 Storing images in the database is generally a bad idea in my opinion. There's much more overhead in retrieving image data from a database then there is in just letting Apache serve up the image. Apache has been highly optimized just for this purpose. Databases generally have not. 14 14 15 15 My suggestion is to just store a URL for the image in the database and write that URL to the dynamic page. Or if you know the path is always going to be the same you could simply store the filename. 16 16 17 -== Robert Walker 17 +== Robert Walker == 18 18 19 19 It seems that different people have different opinions on this topic. I've followed several threads on this and I still haven't come to a conclusion on the best design pattern. 20 20 ... ... @@ -25,17 +25,17 @@ 25 25 An Example: 26 26 Say I have a Product entity and want to upload and store product photos: I would create two entities Product and ProductPhoto. I would then relate them with either a toOne or toMany relationship depending on whether I need one or many ProductPhoto objects for each Product object. 27 27 28 -With this design pattern fetching Product data doesn't directly load the images. Instead EOF will create faults representing the images. 29 -The image data isn't fetched until the fault is fired by accessing the ProductPhoto fault object. So If you fetch 500 Products and batch them into groups of 10 with the [[WODisplayGroup>>Programming__WebObjects-Web Applications-Development-WODisplayGroup]]then your first page would fetch only the first 10 images not the 500 (and only if there is a WOElement? or method that accesses the image data).28 +With this design pattern fetching Product data doesn't directly load the images. Instead EOF will create faults representing the images. 29 +The image data isn't fetched until the fault is fired by accessing the ProductPhoto fault object. So If you fetch 500 Products and batch them into groups of 10 with the WODisplayGroupProgrammingWebObjects-Web Applications-Development-WODisplayGroup then your first page would fetch only the first 10 images not the 500 (and only if there is a WOElement? or method that accesses the image data). 30 30 31 31 This pattern also greatly simplify uploading and storing the images because you can bind the NSData used to upload the image to your ProductPhoto's imageData BLOB. 32 32 33 33 It's probable that many will disagree with me on this issue, but I have had good success, for my purposes, with this design pattern. 34 34 35 -You can find an implementation of this design pattern for both toOne and toMany photos in the JavaRealEstate framework example in 35 +You can find an implementation of this design pattern for both toOne and toMany photos in the JavaRealEstate framework example in 36 36 /Developer/Examples/JavaWebObjects/Frameworks. 37 37 38 -== Michael Halliday 38 +== Michael Halliday == 39 39 40 40 I haven't had any problems storing images in our database (OpenBase). We have developed many "community" based sites with photo albums as well as an online dating service, both use the same methods that Robert talked about in his message. 41 41 ... ... @@ -48,12 +48,12 @@ 48 48 49 49 Again, I know many people will probably disagree with this approach. But, it is working perfectly for us and for dynamic images (or images that the user can change/upload) I think it's the most effective approach. That being said, we do use apache to serve up our static images. 50 50 51 -I'd be interested to hear from others and there experiences with storing images in databases. You hear a lot of people saying "Don't do it, it won't perform well."...but have these people actually tried it? Or have they just been told not to do it. I have been very interested in this topic for a while now and I have done extensive searching but 51 +I'd be interested to hear from others and there experiences with storing images in databases. You hear a lot of people saying "Don't do it, it won't perform well."...but have these people actually tried it? Or have they just been told not to do it. I have been very interested in this topic for a while now and I have done extensive searching but 52 52 never come up with any "correct" answer. I think it also depends on which database you use and how exactly the database itself stores images. I know that some are much better than others and personally this is where you'd most likely run into the performance hit (if any). 53 53 54 -== Geoff Hopson 54 +== Geoff Hopson == 55 55 56 -On the Fortnum & Mason online store http: ~/~/www.fortnumandmason.com, the product catalog is pretty image-heavy. Also, they (F&M) change the catalog and the associated images at least twice a year. So I wrote a tool that allows their product images to be uploaded into the database, simply for the purpose of having everything in a single place for56 +On the Fortnum & Mason online store [[http://www.fortnumandmason.com]], the product catalog is pretty image-heavy. Also, they (F&M) change the catalog and the associated images at least twice a year. So I wrote a tool that allows their product images to be uploaded into the database, simply for the purpose of having everything in a single place for 57 57 backup reasons. When a new catalog is ready to be deployed, the images are extracted from the database and placed under the webserver (since, as everyone notes, webservers are particularly good at vending images). The main F&M web application then gets all it's images from the webserver, as opposed to cached in the webobjects application after a fetch from the database. 58 58 59 59 However, in development, we used the images from the database directly. Command line switch toggles whether the images are read from the webserver or the database. ... ... @@ -60,9 +60,9 @@ 60 60 61 61 Doing all this means that the memory footprint is lower, since the application is not caching images, and it also means that we can do clever things with the webserver to spread the load a little. 62 62 63 -Chuck Hill wrote something on the pros and cons of using the webserver yesterday - http: ~/~/lists.apple.com/mhonarc/webobjects-dev/msg05564.html (use 'archives', 'archives' as the username/password).63 +Chuck Hill wrote something on the pros and cons of using the webserver yesterday - [[http://lists.apple.com/mhonarc/webobjects-dev/msg05564.html]] (use 'archives', 'archives' as the username/password). 64 64 65 -== Arturo Pérez 65 +== Arturo Pérez == 66 66 67 67 My opinion and experience FWIW, having done it both ways. I keep having this discussion so I'd thought I'd put it all down in one place. 68 68 ... ... @@ -92,7 +92,7 @@ 92 92 93 93 Well, my 2 farthings. 94 94 95 -== Chuck Hill 95 +== Chuck Hill == 96 96 97 97 One largish problem with storing them in the database is that EOF will cache the data, at least for a while. For a heavily loaded site or large contents this can really chew up the memory fast. 98 98 ... ... @@ -99,9 +99,9 @@ 99 99 Another alternative is to store them on the file system but not directly available to the web server. Keep an object in the database that references the data on the file system. Use Java streams to move the data from the request to the file system and from the file system into the response. This avoids the EOF overhead but allows your application to control access. It is much more efficient to have the web server directly access and vend the images etc. but if you have access restrictions this is not an option. This hybrid database / file system approach can be useful in that situation. 100 100 101 101 PetiteAbeille wrote about an EOF file system adaptor that may be of interest in relation to this question: 102 -http: ~/~/www.wodeveloper.com/omniLists/eof/2002/June/msg00053.html102 +[[http://www.wodeveloper.com/omniLists/eof/2002/June/msg00053.html]] 103 103 104 -== Tom Pelaia 104 +== Tom Pelaia == 105 105 106 106 We grab images from the database in our WebObjects application (electronic logbook). It is a very heavily accessed site and allows users to make entries that have text, images and other attachments. We have found additional "pros" for loading images from a database. 107 107 ... ... @@ -117,17 +117,17 @@ 117 117 118 118 Whether you choose database storage or filesystem storage really depends on your application. For our application, the electronic logbook is becoming more integrated with other systems and the database has turned out to be critical in that integration. 119 119 120 -== ocs 120 +== ocs == 121 121 122 122 I happily store images in the database, but... my clients use Oracle or FrontBase. The very now though I have the misfortune to work on an application which has to use the MS-SQL thing: seems it really does not support BLOBs well (actually, the database admin just plain told me "do not use a BLOB in your tables, ever~-~-we have the worst experience with them"). 123 123 124 -Myself, I've tried of course :) (with a test database) and found that indeed there seem to be issues, like that a BLOB is never found by a WHERE clause (even if a proper value is provably given). I haven't tested for long :) 124 +Myself, I've tried of course :-) (with a test database) and found that indeed there seem to be issues, like that a BLOB is never found by a WHERE clause (even if a proper value is provably given). I haven't tested for long :-) 125 125 126 -Thus, although I am a strong believer in storing images in the database, I can understand others who are unlucky enough not to be FrontBase users might have different opinions :) 126 +Thus, although I am a strong believer in storing images in the database, I can understand others who are unlucky enough not to be FrontBase users might have different opinions :-) 127 127 128 -The reason I am writing: before deciding where to store your images, do check the concrete database to be used. If FrontBase or Oracle, you probably would want to store them in the database, if MS-SQL, you probably would want to store them in the filesystem :) 128 +The reason I am writing: before deciding where to store your images, do check the concrete database to be used. If FrontBase or Oracle, you probably would want to store them in the database, if MS-SQL, you probably would want to store them in the filesystem :-) 129 129 130 -== Jeff 130 +== Jeff == 131 131 132 132 I'm a little WORusty at the moment, so please excuse any gaffes in this. WO 5.3 has renewed my interest in WebObjects. 133 133 ... ... @@ -135,53 +135,49 @@ 135 135 136 136 Generally, I create a direct action method for dispensing the images. Something like this in your DirectAction (except that you might want to add validation if you don't want just anyone getting to your images by hacking the URL): 137 137 138 -{{ panel}}138 +{{code}} 139 139 140 - public WOActionResults imageAction() 141 - { 142 - // PictureTest is an EOEntity with a BLOB containing the image data 143 - PictureTest pt = getPictureTestEO(); 144 - return jpegResponseWithData(pt.image()); 145 - } 146 - 147 - private PictureTest getPictureTestEO() 148 - { 149 - // Yes - you can get the session in a direct action 150 - // you just need to be prepared to deal with one not existing 151 - // whether you return an image if no session exists depends on 152 - // on your own application needs. 153 - WOSession theSession = existingSession(); 154 - EOEditingContext ec = (theSession == null) ? new EOEditingContext() : theSession.defaultEditingContext(); 155 - String picid = (String)request().formValueForKey("picid"); 156 - return (PictureTest)EOUtilities.objectMatchingKeyAndValue(ec, "PictureTest","id", new Integer(picid)); 157 - } 158 - 159 - private WOResponse jpegResponseWithData(NSData theData) 160 - { 161 - // This method returns the data so that the browser 162 - // recognizes the image type. In this particular application 163 - // I've just hardcoded a mime type of JPEG because I only 164 - // use JPEG images, but a better way would be to store the mime-type 165 - // that corresponds to the image data in the BLOB as a separate 166 - // field. I might revise this sample later on to show that. 167 - WOResponse response = WOApplication.application().createResponseInContext(context()); 168 - response.appendHeader("image/jpeg", "Content-Type"); 169 - response.appendContentData(theData); 170 - return response; 171 - } 140 +public WOActionResults imageAction() { 141 + // PictureTest is an EOEntity with a BLOB containing the image data 142 + PictureTest pt = getPictureTestEO(); 143 + return jpegResponseWithData(pt.image()); 144 +} 172 172 173 -{{/panel}} 146 +private PictureTest getPictureTestEO() { 147 + // Yes - you can get the session in a direct action 148 + // you just need to be prepared to deal with one not existing 149 + // whether you return an image if no session exists depends on 150 + // on your own application needs. 151 + WOSession theSession = existingSession(); 152 + EOEditingContext ec = (theSession == null) ? new EOEditingContext() : theSession.defaultEditingContext(); 153 + String picid = (String)request().formValueForKey("picid"); 154 + return (PictureTest)EOUtilities.objectMatchingKeyAndValue(ec, "PictureTest","id", new Integer(picid)); 155 +} 174 174 157 +private WOResponse jpegResponseWithData(NSData theData) { 158 + // This method returns the data so that the browser 159 + // recognizes the image type. In this particular application 160 + // I've just hardcoded a mime type of JPEG because I only 161 + // use JPEG images, but a better way would be to store the mime-type 162 + // that corresponds to the image data in the BLOB as a separate 163 + // field. I might revise this sample later on to show that. 164 + WOResponse response = WOApplication.application().createResponseInContext(context()); 165 + response.appendHeader("image/jpeg", "Content-Type"); 166 + response.appendContentData(theData); 167 + return response; 168 +} 169 + 170 +{{/code}} 171 + 175 175 Then, in your WOComponent, you create a virtual accessor like this: 176 176 177 -{{ panel}}174 +{{code}} 178 178 179 - public String imageURL() 180 - { 181 - return context().directActionURLForActionNamed("icon", null) + "?picid=" + pictureItem.id(); 182 - } 176 +public String imageURL() { 177 + return context().directActionURLForActionNamed("icon", null) + "?picid=" + pictureItem.id(); 178 +} 183 183 184 -{{/ panel}}180 +{{/code}} 185 185 186 186 in this case, pictureItem is a PictureItem EOEntity instance that I use in a WORepetition - I'm just pulling the id number from the currently selected picture. 187 187 ... ... @@ -189,7 +189,7 @@ 189 189 190 190 This approach eliminated a lot of the overhead that you get by just binding directly to the EOEntity's attribute, and really isn't much extra work. 191 191 192 -== Bill Bumgarner 188 +== Bill Bumgarner == 193 193 194 194 Storing images in the database is generally a bad idea for a whole slew of reasons. First and foremost, it is loads slower than serving images directly from the web server and it completely bypasses numerous automatic "optmiziations" that are present when serving from a filesystem. If it can't be avoided, it can't be avoided.... however, if you have any hopes of scaling your solution to a large community of users or a heavy hit rate, expect to expend a lot of engineering and hardware dollars making images-in-the-database go fast. 195 195 ... ... @@ -198,7 +198,7 @@ 198 198 Images are normally served statically from the filesystem... because you are now serving them as dynamic content, the following performance hits occur: 199 199 200 200 * no client side caching [[ouch]] five copies of a single image on a page yields five seperate hits on WO. 201 -* image requests must be serialized ~-~-not only do IMAGE hits have to be serialized, but all other hits on the WOF app will have to wait for any pending image hits to be handled. In terms of Netscape's REALLY SLOW table layout algorithm that requires the size of all images to be known, this means that the user WON'T see the contents of the table until ALL image hits have returned at least the size of the image.... since hits are serialized, that means that all but the last image must be entirely handled.197 +* image requests must be serialized- not only do IMAGE hits have to be serialized, but all other hits on the WOF app will have to wait for any pending image hits to be handled. In terms of Netscape's REALLY SLOW table layout algorithm that requires the size of all images to be known, this means that the user WON'T see the contents of the table until ALL image hits have returned at least the size of the image.... since hits are serialized, that means that all but the last image must be entirely handled. 202 202 * performance difference between a static hit vs. a fully dynamic hit is tremendous [[in favor of static]]. Think about it... a static hit basically means the web server opens a file, reads/writes the contents to a socket, closes... dynamic hits require IPC, a database round trip [[maybe]], a bunch of memory munging, a pass through request/response, etc.etc.etc... 203 203 * no server side caching; every instance of your app will end up with a copy of every image served in its memory. As well, the IPC between database and WO app server will have to pass all that data back and forth, as well. 204 204 * most databases are not designed to handle BLOBs well.... regardless ... ... @@ -208,6 +208,4 @@ 208 208 Stick a path in the filesystem in your database instead of a blob; abstract arbitrarily to facilitate administration, etc... 209 209 If you REALLY need the images to come from the database, build an image manager that maintains a hierarchy of the images in the filesystem and arbitrates the updates between the database and the images. 210 210 211 -One thought; if an image needs to be refreshed and you are worried about client-side or proxying-firewall caching, rename the image in the filesystem (or move it) and generate a new URL~-~- this should be the image managers responsibility. 212 - 213 -Category:WebObjects 207 +One thought; if an image needs to be refreshed and you are worried about client-side or proxying-firewall caching, rename the image in the filesystem (or move it) and generate a new URL- this should be the image managers responsibility.