Custom Python App on EMR Serverless

Environment

• Python 3.9.9
• EMR Serverless: 6.13
• TensorFlow: 2.11

Reference

• https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/using-python-libraries.html
• https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-spark.html
• https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/using-python.html

I had to jump through a few hoops to get a PySpark application running on EMR Serverless. Below are the steps I followed, along with final functioning configuration, and at the bottom of this post is a few errors I encountered along the way.

Steps

1. Setup Build Environment

For a packaged application to work it must be built in an environment very similar to that of EMR Serverless; specifically, Amazon Linux 2. Plenty of mention is made online about using platform=linux/arm64 amazonlinux:2 to achieve this in Docker. I could not get this to work – when attempting to output the image it would just hang forever – I suspect because I’m on OSX, so I ended up spinning up an EC2 instance for my build environment, based on Amazon Linux 2 image.

2. Setup and Run Build Script

Mine was almost identical to the one found here, just with a few tweaks. Place your project requirements.txt in your build environment working directory and:

sudo yum install -y gcc openssl-devel bzip2-devel libffi-devel tar gzip wget make xz-devel lzma

wget https://www.python.org/ftp/python/3.9.9/Python-3.9.9.tgz && \
    tar xzf Python-3.9.9.tgz && \
    cd Python-3.9.9  && \
    ./configure --enable-optimizations && \
    sudo make altinstall

sudo yum install -y python3-pip

# Create python venv with Python 3.9.9
python3.9 -m venv pyspark_venv_python_3.9.9 --copies

# copy system python3 libraries to venv
cp -r /usr/local/lib/python3.9/* ./pyspark_venv_python_3.9.9/lib/python3.9/

# package venv to archive.
# **Note** that you have to supply --python-prefix option
# to make sure python starts with the path where your
# copied libraries are present.
# Copying the python binary to the "environment" directory.
source pyspark_venv_python_3.9.9/bin/activate && \
    pip install venv-pack && \
    pip install -r requirements.txt

sudo mkdir -p /home/hadoop/environment
source pyspark_venv_python_3.9.9/bin/activate &&  \
    venv-pack -f -o pyspark_venv_python_3.9.9.tar.gz --python-prefix /home/hadoop/environment

# You'll need to reference this path/file in your EMR Serverless job config
aws s3 cp pyspark_venv_python_3.9.9.tar.gz s3://<path_to>/<project_artifacts>/

3. Align Python Lib with EMR Requirements

If you’re smart, you started out with EMR-capable lib versions and worked backward from there. If, like me, you were handed a project where this was not the case, you’ll likely have to backoff dependency versions to make them compatible with EMR Serverless.

4. Zip and Upload Custom Python Modules

• From directory containing your application code: zip -r my_custom_modules my_custom_modules/
• aws s3 cp my_custom_modules.zip s3://<path_to>/<project_artifacts>/my_custom_modules.zip

4. Configure EMR Serverless

• In EMR Studio, create an application. I allowed it create and use a default IAM role.
• Upload your entry point script to S3 and define it under ‘Script location’.
• Add any script arguments you need to pass (and your app is prepared to parse).
• I landed on the following Spark properties in order to get the job to run:
  --conf spark.archives=s3://<path_to>/<project_artifacts>/pyspark_venv_python_3.9.9.tar.gz#environment
--conf spark.emr-serverless.driverEnv.PYSPARK_PYTHON=./environment/bin/python
--conf spark.emr-serverless.driverEnv.PYSPARK_DRIVER_PYTHON=./environment/bin/python
--conf spark.executorEnv.PYSPARK_PYTHON=./environment/bin/python
--conf spark.submit.pyFiles=s3://<path_to>/<project_artifacts>/my_custom_modules.zip
--conf spark.files=s3://<path_to>/<project_artifacts>/some_other_file.yml, s3://<path_to>/<project_artifacts>/second_other_file.yml

Errors

Encountered along the way.

Error	Resolution
Traceback (most recent call last): File "/home/hadoop/environment/lib/python3.9/site-packages/fastavro/read.py", line 2, in <module> from . import _read File "fastavro/_read.pyx", line 11, in init fastavro._read File "/home/hadoop/environment/lib/python3.9/lzma.py", line 27, in <module> from _lzma import * ModuleNotFoundError: No module named '_lzma'	In build environment: sudo yum install lzma
ModuleNotFoundError: No module named '<my custom modules>'	These steps, from the base of your application code: * zip -r my_custom_modules.zip my_custom_modules/ * upload zip file to s3 bucket * add to your job spark properties: --conf spark.submit.pyFiles=s3://<path_to>/my_custom_modules.zip
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with...	Downgraded to urllib3=1.26.6
ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/home/hadoop/environment/lib/python3.9/site-packages/google/protobuf/internal/__init__.py)	Fetch latest version – e.g., from fully-updated installation – of protobuf’s ‘builder.py’ to your project’s Python packages at Lib/site-packages/google/protobuff/internal. See here for details.
Docker build env: “copying <n> files…” Never finishes.	I had to abandon a Dockerized Amazon Linux 2 build environment, I suspect it had something to do with my Apple silicon. I ended up spinning up a VM on AWS and using their Amazon Linux 2.

Pass HTTP Headers with Non Proxy Lambda Integration in AWS API Gateway

I set out to pass an HTTP header through API Gateway by mapping it in the method and integration request configurations (specifically using Serverless framework/template), based on various documentation I found online indicating I should do so. While troubleshooting, I at one point removed the mappings entirely and noticed that it *just worked*.

I.e., with no configuration in the method or integration request mappings, the HTTP header of interest (in this case, Authorization) was passed through API Gateway to my Lambda and accessible in the event object @ event[‘headers’][‘Authorization’]. I have seen no mention of this online, but perhaps it was quietly added by AWS at some point.

Not sure if anyone else has run into this…

Limiting User to SFTP for Uploading Web Content

I required the following:

• System user that could upload content to a directory in root web directory (default root: /var/www/html)
• Limit user from interactive SSH
• Limit user from other areas of OS

Specifically, I am working within the AWS distribution on a hosted EC2 instance.

I found posts online that accomplished part of what I needed. But my steps to achieving this were:

1. Create the user. In my case, user webpub. This creates an entry in /etc/passwd as well as a home directory under /home: 

   sudo useradd webpub

2. These next few steps I found here. Create a ‘jail’ directory that we will constrain the user. I created it in /var.

   sudo mkdir /var/jail
   

3. An important note is that the jail directory and all directories beneath it must be owned by user root in order for the Chroot declaration to work. If you get setup and notice that you are correctly authenticating but then the connection immediately drops, this could be your problem. Now create a sub-directory that will serve as the access point for the user to the web content:

   sudo mkdir /var/jail/www

4. The directory created above can also be owned by root. Create a sub-directory under web content root that we will restrict this user to. In this case, the same name as the user:

   sudo mkdir /var/www/html/webpub

5. The directory created above can also be owned by root. Now create the link between the jail and the content directory by binding the two:

   sudo mount -o bind /var/www/html/webpub /var/jail/www

6. In /etc/passwd, update the user webpub‘s home directory (where they will land upon logging in) to /var/jail/www.
7. Update /etc/ssh/sshd_config to jail the user upon logging in. Start by commenting the line Subsystem sftp /usr/libexec/openssh/sftp-server and then adding configuration for the internal-sftp sub-system. When done, it will look like (commented line and all): 

   #Subsystem sftp /usr/libexec/openssh/sftp-server
   Subsystem sftp internal-sftp
   Match User webpub
           ChrootDirectory /var/jail
           ForceCommand internal-sftp
           X11Forwarding no
           AllowTcpForwarding no

8. The ChrootDirectory jails the user while ForceCommand internal-sftp lists the user to only being able to login via SFTP. Now restart openssh:

   sudo /etc/init.d/sshd restart

9. In my setup, I have password authentication disabled, so the last step is create a private/public key pair and install client/server side. Remember that authorized_keys (and its parent directory .ssh) must reside in the home directory for webpub, which we set earlier as /var/jail/www. Since that directory is bound to /var/www/html/webpub, though, these artifacts reside in the latter directory.

Gaming System Builds (~$500 and ~$1000)

Recently, a couple of friends have tapped me (or did I volunteer?) to spec out parts for a new gaming rig. The first friend was looking in the $500-600 range in order to get his League of Legends on, the second wants to replace his aging PC before the WoW expansions drops in a week or two. I figured I would capture here what I came up with.

---

The $500 (oh, okay, “sub-$600”) gaming rig.

This did prove a little challenging. The price point is low enough where some serious consideration has to be given to where to cut corners and still outfit what can be considered a complete PC. Admittedly, I assembled this list a few months ago, so prices may have dropped and “best value” components shifted a bit since then (gotta love technology).

Motherboard: ASUS M5A97 R2.0 Socket AM3+ ATX ($90)

CPU: AMD FX-6300 Vishera 6-Core 3.5GHz (4.1GHz Turbo) Socket AM3+ ($110)

Video Card: EVGA 02G-P4-2742-KR GeForce GT 740 Superclocked 2GB 128-Bit DDR3 ($90)

Memory: CORSAIR Vengeance 8GB 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) ($80)

Power Supply: CORSAIR CX series CX600 600W ATX12V v2.3 ($80)

Hard Drive: Seagate Barracuda ST1000DM003 1TB 7200 RPM 64MB Cache SATA ($55)

Optical Drive: Asus or Samsung ($20)

Case: Antec Three Hundred ($65)

Total cost: $590

 

Upgrades that could be made to the above:

Video Card: EVGA 03G-P4-2667-KR G-SYNC Support GeForce GTX 660 FTW Signature 2 3GB 192-bit GDDR5 (+$90)

CPU/Motherboard: Upgrade to Intel i5 (+$150)

New total cost: $830

 

Downgrades that could be made to above:

Motherboard: Asus to MSI (-$20)

PSU: 600w to 500w PSU (-$15)

Memory: 8GB to 4GB (-$40)

New total cost: $515

 

---

The $1k gaming rig.

A little more breathing room here, but (and it’s a big ‘but’) this particular friend has his sights set on a Core i7. There goes about a third of the budget.

He was also looking at a pre-built (some great values to be had here) system, the ASUS M32AD-US032S Desktop PC, selling for $969, which comes with the following specs:

Intel Core i7 4790 (3.6GHz)

Chipset: Intel H81

16GB DDR3 2TB HDD

Windows 8.1 64-Bit

NVIDIA GeForce GT 740 4 GB

300W PSU

I sought to come up with a similarly priced alternative that might be more tuned to the discerning builder/gamer. A few notes driving my decision-making:

• The box above is put together by Asus, and Asus knows what it’s doing. I’m fairly certain it’s going to run your games just fine, and right out of the box, no less. That said…
• 300w struck me as a borderline. Again, I’m sure the PC is going to run fine, but how about a little overhead for those future upgrades?
• I couldn’t find much information on the specific components actually used…I’m going to go ahead and venture they’ll be mainly Asus, but who knows. When *I* build a system, though, I _do_ know.
• There is some real value here to those who need a Windows license, which are running north of $100 a pop right now. I disregard such license in my builds, but if you need one, that’s $100 right off the bat.
• Input devices. It’s not much of a consideration for me – I like to latch onto my own – but the Asus prebuilt comes with keyboard and mouse.

My answer to the Asus pre-built:

CPU: Intel Core i7-4790 Haswell Quad-Core 3.6GHz LGA 1150 ($310)

Video Card: EVGA 03G-P4-2667-KR G-SYNC Support GeForce GTX 660 FTW Signature 2 3GB 192-bit GDDR5 ($180)

Motherboard: ASUS Z97-A LGA 1150 Intel Z97 ($150)

Memory: CORSAIR Vengeance 8GB 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) ($80)

Power Supply: CORSAIR CX series CX600 600W ATX12V v2.3 ($80)

Hard Drive: Seagate Hybrid Drive ST1000DX001 1TB MLC/8GB 64MB Cache SATA ($80)

Case: Antec Nine Hundred ($95) or Antec Three Hundred ($65)

Total cost: $975

The above gets you into features offered by the Z97 chipset that the H81 does not have. I also give the video card a pretty serious bump. I cut corners with memory, going from 16GB to 8GB. Some people will scream about this, but 8GB is going to be fine right now and RAM is a straightforward upgrade down the line. I spec a robust power supply with room to grow, and an accompanying big, cool, quiet Antec Nine Hundred. I sacrificed some storage in exchange for the speed benefits of Hybrid. If you’re hoarding media, that might prove unpalatable, but I might also recommend going out and getting a giant, slow(er) drive for such things (unless you’re doing a bunch of editing of said media, etc., in which case you’re peripheral to my target audience, anyway).

---

 

As usual, I’m amazed at the caliber of hardware that can be gotten for a reasonable price. I put a rig together about four years ago, in that ~$1k range, and it still goes strong with WoW (the only game I still really play, on occasion) cranked. The more demanding games, running at higher resolutions (1080p widescreen, etc.) than what I’m running would make it sweat, I’m sure, but my general point is that you can reasonably expect to get some quality time from a system in this range. Even the “sub-$600” system offers some upgrade paths that will keep you chugging for a bit.

Asus RT-AC68R Periodically Drops Wireless (2.4GHz) Connection

This does not strike me as specific only to this router, but has to do with a configuration setting around 20MHz vs. 40MHz channel bandwidth.

I picked up an Asus RT-AC68R a while back and it’s been fantastic. One bit of behavior I wasn’t crazy about was the periodic – and somewhat frequent – dropping of the wireless connection. This happened with multiple devices (all of which happened to be Apple). I read around a bit online and there were a few suggestions, ranging from configuration to third-party firmware, but the one I homed in on had to do with channel bandwidth. I live in an urban environment and there are many APs within range of me. Evidently, by default, the Asus comes configured with 2.4GHz to use a channel bandwidth of 40MHz (or 20/40) as opposed to a strict 20MHz bandwidth, and this has been identified as a potential issue in crowded AP environments. I limited the channel bandwidth to the more concise 20MHz and my dropped connections have disappeared in the weeks since.

Logitech Harmony 676 Universal Remote

Lately, I have felt the temptation to purchase a remote with which I could rule over the myriad components at our house. A couple of weeks ago this desire was augmented by the all-too-frequent itch to buy a new toy. And so I present to you Logitech’s Harmony 676 Universal Remote.

I have followed Harmony’s (acquired by Logitech) line of remotes since probably their inception. They have been generally well-received as affordable alternatives to high-end system controllers. I initially narrowed my choices down to the Logitech Harmony 676 and the 680. After much reading around, I arrived at the conclusion that the only real difference was button layout, and I was partial to the button layout on the 676. The capabilities of the two remotes seem to be about on par with one another. I had a few criteria in mind that I wanted to see in any remote I purchased:

1. Reasonable Price
I don’t mind dropping some money on a decent remote, but I have no desire to part with hundreds of dollars, much less thousands.

2. Traditional Formfactor
By this, I mean the shape and size of conventional remotes with actual “hard” buttons. I absolutely did not want some beast of a remote that requires two hands to operate. I also am not a fan of the giant LCD screen that allows you to layout “soft” buttons on it. The tactile response is non-existent and provides me no way of feeling my way around in the dark, should I ever have to. In my mind, I was picturing a remote with a small LCD screen to provide feedback or navigation, but not a large, power-hungry one.

3. Ergonomics
This includes a few things. First, I wanted a remote that is fairly easy to configure yet powerful enough to do a bit beyond the basic functions for each component. Second, I don’t want to be constantly changing batteries, so they’d better last a while. Third, my ideal remote would fit comfortably in one hand and have a backlight that I can easily kick on, if needed. Lastly, I wouldn’t mind it actually looking somewhat good sitting on the coffee table! 

Price

Sure, you can go to Walmart and pick up a “universal” remote for $20. Okay, and maybe Aldi’s has one for $10. But all-in-all, I think the $72 I paid for the Logitech Harmony 676 is a fantastic price, given its capabilities. The 676 was available from Newegg.com for $122 less a $50 mail-in rebate. I have seen similar prices at other online retailers, but I happen to {heart} the ‘egg (it’s just such a cute name). At any rate, this remote’s capabilities are on par with remotes costing 10 times as much. 

First Impression

The freakin’ thief-deterrent plastic always leaves a sour first impression of so many electronic products these days. The first five minutes with my remote consisted of me trying to liberate it from its enclosure without having myself a bloodletting. Oh, I got to it, though:

 

The 676 has a very comfortable, almost rubber-like plastic grip wrapping the back and sides. The faceplate is handsome enough, in my opinion: techy looking but in an unobtrusive kind of way. Its silver in color but the 676 comes with red and blue replacements (I’m not a big fan of the color faceplate thing, but maybe some night when I’m feeling sassy…). The silver is a good look, though. The buttons are laid out with space enough in between them (as much as you’ll find on a remote designed to do so much, anyway) so you can somewhat feel your way around, at least to some of the more major functions. A smallish LCD screen with room for about four lines of text stares up at you, and three buttons on either side provide access for making selections from the menu on the screen. The shape of the remote is very comfortably contoured. In short: it had me at “hello.”

There are 4 buttons at the top of the remote that are colored. These grant access to the remote’s preprogrammed “activities” functions. By default, there is one button each for watching a movie, watching TV, and listening to music. The fourth colored button gives access to additional activities through use of the LCD screen. These four colored buttons will become your new little friends. 

Setup

As you may have heard, this is where the Logitech Harmony line of remotes really shine. It was almost disgusting how easy setup was; I felt guilty. From the moment you turn the remote on, it’s holding your hand through use of its LCD screen. Also, the 676 comes with four Duracell AAA batteries. Trivial, perhaps, but a nice touch. A USB cable is also provided.

Yes, this remote must be hooked up via USB to an Internet-ready computer in order to be properly programmed, but that doesn’t bother me. Perhaps you’re of a different mindset (e.g. damn convergence to hell!). I installed the software without incident and made a login for myself when it took me to Logitech’s site. The online check noticed that the software and firmware were behind so it walked me through the updates. It took several minutes, but no big deal; you’ll earn those minutes back during the configuration. The online web configuration then asked me to input the brand and model of all the components in my A/V system. And my, what a database Logitech keeps! Everything from receivers to TV’s to PS2’s to Media PC’s to motorized shades and dimmable lights. My understanding is that if your hardware isn’t on there, contact Logitech and they’ll custom map a setup for you. From what I saw, though, it’s a pretty thorough list.

I entered the information for each of our components, suddenly realizing that I really need to latch onto some motorized shades. Once all components are added to the list, the online configuration walks you through the setup of your “activities.” For example, in order to watch a movie, the setup will ask you which components you want involved in the activity (the rest will be off). It also prompts you for which input each component should be set to and which component should be controlling the volume, as well as more detailed settings such as what effects to implement on the A/V receiver. The setup is extremely intuitive, even for a newbie, though some running back and forth between computer and A/V system will likely be required. Luckily, our A/V system happens to be right by the beer fridge, so this worked out just fine for me.

The basic setup for each activity is simple. You can, if you want, get into extensive detailing and tweaking for each activity and/or component. I haven’t delved into this too much yet (the default setup is just THAT good), but likely will in the future, at least to an extent.

Once the activities and components are configured, you save the settings to the flash memory (retained even if batteries are removed) in the remote. This will take a few minutes. The little LCD screen on the remote keeps you apprised of what’s going on and tells you when it is finished. Unplug it and you are ready to go. 

Ease of Use

I was almost nervous when I walked in the living room and hit the “Watch Movie” button. I just wasn’t sure this thing was going to work. But it did. Everything fired up exactly as it should have on exactly the right inputs. I couldn’t help but smile as I looked with disdain at the three remotes sitting below me. Their time was up.

Naturally, I had to play with those activity buttons for about half an hour before looking at anything else. The 676 will even prompt you when you hit an activity button, asking you if everything is working okay. If you say no, it will try to correct the situation. I ran into this immediately when I tagged the “Watch TV” button. Our cable box must have already been on and it turned it off when firing off the macro. When it asked if everything was functioning correctly and I said no, it turned the cable box on and and the cable signal appeared on the TV. My smile faltered slightly because, well, that was kind of spooky; I have no idea how it knew the cable box was the component out of sync. Looking down at the LCD screen, I saw that it was asking me if that fixed it. I hesitantly replied in the affirmative without making too much eye contact.

The activity buttons are cool, I have to admit. Some confusion can arise if components get out of sync, though. The remote has no way of knowing if a component is on or off. Some components (especially newer ones) have discrete signals for ON and OFF. Those that do not, however, can find themselves being turned on or off when really the opposite was desired, depending on their existing state. I guess it’s possible that the 676 knows which components have discrete on/off signals and is thus able to better troubleshoot when things get out of sync. But still…I won’t be doublecrossing the 676 anytime soon.

I was pleasantly surprised to find that the default layouts for component controls are handled with aplomb by the 676. It nimbly managed the navigational and informational features of the cable box, assigning an intuitive layout for such functions. In addition to the functions assigned to the hard buttons, more obscure commands are accessible through the LCD screen and its associated buttons. Using this command set, I found the remote was able to do everything from calibrating the picture on our Sony HDTV to adjusting speaker level and distance compensation on the Yamaha receiver. Very cool.

I do have an issue with the remote when it comes to some of the more obscure commands. The labeling of the commands on the LCD screen can become someone crammed and difficult to read, at times. Furthermore, it seems as if a few of the commands are not set-up correctly (e.g. the command for invoking the “Jazz” surround effect actually triggers “Rock Concert”). It appears as if most of the options are there to control the finer points of the receiver but that a few of them might be mixed up with one another. In the end, I take this as a pretty small issue. It is also possible that the assignments can be manually reconfigured; I have not had a chance to dive into that yet.

Accessing the more detailed component functions involves switching out of the “activity” mode and selecting the specific component to control. This is made easy by the “Device” button. Pressing this button brings up a list of all programmed components on the LCD screen. Selecting a component on this list remaps the remote for detailed control over that particular peripheral. Once any tweaking has been taken care of, simply hit the “Device” button to switch back to the activity layout that was last activated. Again: very cool.

The Logitech Harmony 676 is capable of controlling media center PCs. I have not had the chance to add an infrared receiver to our media center PC, but I plan to at some point. There is a “Media” button on the 676 that supposedly brings up the access screen for all media on the PC. It sounds intriguing, but I have not had any experience with it, as of yet.

Physically, this remote is a beauty. It has a solid, comfortable feel to it. Its elegant contours fit naturally in my hand. The buttons have a crisp, snappy feedback, too, and I notice very little of the lag that can come with using a universal remote with various equipment. After using the remote for only a little while, I have become accustomed to the placement of some of the more heavily used buttons and can find them without looking. The “Glow” button is easy enough to find in the dark and provides a pleasant blue backlight to all buttons and the LCD screen. 

Conclusion

The Logitech Harmony 676 Remote is the remote that I have been waiting for. It manages to blend ease of use for everyday functions with brute power for those times you wish to delve into the detailed settings of a component. And it looks good while doing it. Appropriate for the neophyte as well as harder core A/V junkies, it comes at a price that won’t leave too big of a dent in your wallet. While there are some wrinkles with some of the detailed function assignments, overall the 676 performed marvelously and its activity modes make full system control a snap.

Setup: A
Ease of Use: B+
Physical Attributes: A

Rear Projection Televisions: LCD vs DLP

A friend of mine recently emailed me the following question:

“We are looking at getting a TV and I was wondering what your thoughts are about TVs. We were looking into about a 42″ or 50″ HDTV. We were looking at the LCD Sony and also the Samsung with the DPL stuff. We only want to spend 2,000. We are in no hurry to get a TV and since everything is going to HD should we wait a year or so? Are prices going to drop soon?”

I replied to his email and then realized that it might be informative to others out there. My response follows:

I’m assuming by “LCD Sony” you mean an LCD rear-projection television and not a true LCD. LCD rear projectors are like regular rear projection televisions, the obvious difference being that an LCD provides the source image as opposed to a traditional CRT. LCD rear projectors are quickly becoming the de facto standard for rear projectors. DLP is another breed of rear projector.

So I’m going to operate under the assumption you are looking at LCD and DLP rear projectors (a true “flat panel” LCD of the size you’re talking would be big bucks). Rear projection sets are where it’s at these days when it comes to bang for buck, and the LCD and DLP models offer cabinets that aren’t as deep and awkward as the cabinets of yesterday’s CRT rear projectors.

You’ve picked a great time to by a television. Prices have plummeted and there are some excellent sets in the $2,000 range and below. Sure, prices will continue to steadily drop with the various TV technologies, but such will always be the case. As for high-definition, there is no reason to wait. HD is here and there are plenty of sets that handle HD signals with aplomb. And it doesn’t take $3,000 (or even necessarily $2,000) to reap the benefits, though your $2,000 figure is a reasonable one to start with and opens up plenty of possibilities.

As it turns out, you’ve picked two of my favorite brands. Though there have been times when Consumer Reports has given Sony sub-par marks on maintenance history, it seems like over the years they have been pretty consistent in delivering great pictures, from their Trinitron tubes to their upper line WEGA sets. Then you have Samsung, who has transformed their image from low-budget, brand-name-alternative to industry powerhouse with cutting edge technology at reasonable prices (there are rumors that Samsung is going to begin nudging prices up now that they have secured a substantial market share…dunno how much credence there is to that). I am a Samsung fan. For televisions, I’d have to seriously consider Sony, too, though.

As for the two technologies you’re looking at, as a general statement I think I prefer DLP RPTV’s (rear projection TV’s) over LCD RPTV’s. DLP has ironed out *most* of the wrinkles in its technology and it really can deliver. Probably the best set I’ve calibrated was a Samsung DLP. We watched Finding Nemo on it afterward and it was almost disgusting how good it looked (progressive scan DVD player will be a must for you, by the way…but that’s pocket change; most players are coming with progressive scan these days). Anyway, each technology does have its weaknesses, though each are capable of delivering an excellent picture…let me briefly breakdown the most notable…

LCD RPTV’s: A couple of things to look for on the sets, though. Sometimes these sets can exhibit a “screen door” effect since LCD sets, by design, demand a bit more room between the pixels on the LCD. I would guess most sets you will look at won’t have this problem, but it’s worth keeping in mind. DLP doesn’t have this issue. Also, the most critical aspect of LCD RPTV’s is contrast ratio. They have traditionally had a difficult time displaying “true” blacks and subtle variations in grays (accurate grayscale reproduction is vital for accurate color rendition). The latest LCD’s, though, have really come a long way in overcoming this. But again, it’s worth looking for. This can vary greatly by model/manufacturer.

DLP RPTV’s: Lately, the main culprit with this technology is what they call a “rainbow effect,” though the problem has been mitigated in modern sets. DLP sets use a single color wheel through which passes all three colors sequentially (Red, Green, Blue). This can lead to brief, almost imperceptible streaks of color on some sets. If it’s present, people typically only notice it when they’re quickly moving their eyes across the screen. It sounds like a strange phenomenon; personally, I’ve never experienced it…then again, I haven’t been watching too many DLP’s. DLP’s tend to produce great blacks. There was a time when they had issues mustering ample amounts of brightness but I think that’s pretty much a thing of the past. It’s also been said that DLP is prone to a bit more “video noise” than other rear projection technologies. DLP sets also tend to be more expensive.

Both types will require the replacement of the lamp periodically. And when I say periodically, I mean YEARS. The lamps have thousands of hours on them. Neither type of set is prone to the “burn-in” that you used to hear so much about with CRT’s. One thing that is a must if you buy a television of this caliber: CALIBRATE IT! Ideally, you’d drop a couple hundred dollars and have a certified tech come out, access the service menu, calibrate it, then save the settings. The next best thing, however, is to do it yourself. Though you can’t access the service menu, the conventional adjustments allow you to tweak it fairly well. Basic test patterns that can be used for calibration are on any movie DVD that is labeled as “THX”. They can be accessed through the DVD’s menu.

There can be a dramatic difference between a calibrated and non-calibrated set. I could give you a hand with this, if the time should come. It’s not that difficult. The main thing that will almost certainly need to be adjusted is brightness. They are usually jacked WAY up from the factory. That’s why looking at TV’s in a store is usually pretty worthless; you rarely get much of an idea of the television’s true potential compared to those around it. Manufacturers know that our eyes tend to favor brighter pictures, so that’s how they ship them. Unfortunately, brighter is not better…it comes at the expense of things like color reproduction and contrast.

Sorry to throw all this stuff at you, but it’s good background information to have. As for specific Sony and Samsung sets, let me get back to you after doing some looking around. My instinct says that either one would be a fine purchase (though I’d probably lead toward the Samsung), but I’ve gotten behind in reading some of my periodicals…I’ll bet there’s a review or two that could come in handy. Until then, do some looking around on the Internet. When you DO buy, make sure you get it from a place with a decent return policy. And before I bought I’d consider taking along a THX-certified DVD and at least *somewhat* calibrating the set right there in the store (not the best with glaring fluorescent lights overhead, I know). They should have no qualms with you doing that. Then get right up on the set and look for anything suspicious. How do the blacks look? Is there any strange artifacting? The other thing that is kind of a bummer is the video signal fed to the TV’s in stores such as Best Buy, etc. are horrible. So you really won’t get an accurate idea until you get the set home. More upscale audio/video establishments usually have better set-ups: calibrated TV’s, dimmed lighting, etc.

All this being said, there are a bunch of great TV’s out there. And these days, the differences in quality are becoming fewer and more subtle, often residing as much between different makes and models as which technology is used. You picked a great time to buy a TV.

Freeware / Open Source Applications

I have decided to compile here a list of my favorite freeware/open source applications. For simplicity’s sake, I will limit the list to that software running on Windows since it remains the most prominent operating system.

You might be surprised to find there is a lot of freely available software that is as good, and sometime BETTER, than its commercial counterparts. No cost, no ads, no nagging (of course, consider donating to that software you find useful!). It’s a beautiful thing. Keep in mind, there are others out there; these just happen to be my favorites. I will likely revise this list in the future as I stumble upon new applications. Without further ado:

Office Suite: Open Office
Archive Utility: Zip Genius
Desktop Post-It Notes: AT Notes
Flow Charts: Dia
Media PC: Media Portal

Connectivity
Web Browser: Mozilla Firefox
Email Client: Mozilla Thunderbird
FTP Client: FileZilla (Firefox also has a very decent FTP extension)
SCP/SFTP Client: WinSCP
SSH Client: Putty
Instant Messaging (ICQ, AOL, MSN, etc.): Gaim
Peer-to-Peer File Sharing: eMule
BitTorrent Client: Yet Another BitTorrent Client
Telephony (VoIP): Skype
Multi-Platform Desktop Control: Real VNC
Port Scanning: Nmap

Audio/Video
Audio Editing: Audacity
Audio Encoding/CD-Ripping: Exact Audio Copy (EAC)
MP3 Stream Ripper: Stream Ripper
Video Encoding/Processing: Virtual Dub
Music/Video Player: Winamp
DVD Player: VideoLAN
Quicktime: Quicktime Alternative
RealPlayer: RealPlayer Alternative

Graphics
Viewing/Converting Images: Irfanview
Image Editing: The Gimp
Image Viewing/Organizing: Picasa
PDF Viewer: Fox It PDF Reader
PDF Creator: PDF Creator
Web Authoring: Nvu
RGB Color Calculator/Matcher: Easy RGB

CD-DVD Software
Windows XP SlipStreaming Utility: Autostreamer
Display Audio CD Tracks as .wav Files in Explorer: CDFS.vxd
CD-R Identifier: CD Media Code Identifier
VideoCD: VCDgear
CD/DVD Burner: CD Burner XP
Video-DVD Ripping: DVD Decrypter
Video-DVD Copying: DVD Shrink
DVD-R Identifier: DVD Identifier
ISO Utility: ISO Buster
Virtual Drives: Daemon Tools

System Maintenance
Anti-Spyware: Microsoft AntiSpyware and Spybot
Anti-Virus: AVG Free Edition
File Backup Utility: SyncBack Freeware and Allway Sync
Hard Drive Imaging: SelfImage
Floppy Imaging: RawWrite
Hard Drive Fitness: IBM Hard Drive Fitness Test
Disk and Data Recovery: ResQPro
Restore Deleted Files: Restoration

Tweaking and Miscellany
System Information: Belarc Advisor and CPU-Z
Benchmarking, 3D Peformance: 3DMark
Benchmarking, Application Performance: PCMark
Benchmarking, Hard Drive: HD Tach
Hard Drive Temperature: Hard Drive Thermometer
Monitor and Control Motherboard: Motherboard Monitor
Memory Diagnostic: MemTest86
Boot Analyzer/Cleaner: BootVIS
Disk Wiper: Darik’s Boot and Nuke
Partition Manager: Ranish
Windows Process Management: Itt Bitty Process Manager
File Encryption: Cypherix
Tweak Windows XP: TweakUI
Find Out What Program is Locking a File: WhoLockMe?

For All Those Times You’re Using Your Computer Outside: Anti-Mosquito

And don’t forget the ultimate open source software: Linux and it’s myriad components. Feel like taking the plunge? I’d recommend Ubuntu. It’s a cinch to get up and running.